By Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor Cărbune, Michelle Casbon, Mayank Chaturvedi, Aditya Chawla, Victor Cotruta, Alice Coucke, Phil Culliton, Robert Dadashi, Lucas Dixon, Mohamed Elhawaty, Utku Evci, Clément Farabet, Johan Ferret, Filippo Galgani, Sertan Girgin, Jean-Bastien Grill, Maarten Grootendorst, Jiaxian Guo, Cassidy Hardin, Yanzhang He, Steven M. Hernandez, Omri Homburger, Léonard Hussenot, Juyeong Ji, Armand Joulin, Aishwarya Kamath, Parnian Kassraie, Olivier Lacombe, Preethi Lahoti, Gaël Liu, Gus Martins, Luciano Martins, Tatiana Matejovicova, Ramona Merhej, Nikola Momchev, Sneha Mondal, Ryan Mullins, Sindhu Raghuram Panyam, Shreya Pathak, Sarah Perrin, André Susano Pinto, Etienne Pot, Angéline Pouget, Alexandre Ramé, Sabela Ramos, Douglas Reid, David Rim, Morgane Rivière, Karsten Roth, Louis Rouillard, Omar Sanseviero, Pier Giuseppe Sessa, Shane Settle, Danila Sinopalnikov, Sara Smoot, Piotr Stanczyk, Andreas Steiner, Lawrence Stewart, Ilya Tolstikhin, Michael Tschannen, Anton Tsitsulin, Nino Vieillard, Renjie Wu, Pingmei Xu, Haichuan Yang, Edouard Yvinec, Biao Zhang, Li Zhang, Joe Zou, Nicolas Aagnes, Abdelrahman Abdelhamed, Jakub Adamek, Shivani Agrawal, Shubham Agrawal, Ibrahim Alabdulmohsin, Jean Baptiste Alayrac, Uri Alon, Chandramouli Amarnath, Ankesh Anand, Chrysovalantis Anastasiou, Setareh Ariafar, François-Xavier Aubet, Kyriakos Axiotis, Federico Barbero, Joelle Barral, Alexei Bendebury, Urs Bergmann, Stanley Bileschi, Kat Black, Mathieu Blondel, Sebastian Borgeaud, Arthur Bražinskas, Ryan Burnell, Robert Busa-Fekete, Mu Cai, Daniele Calandriello, Glenn Cameron, Charlotte Caucheteux, Rahma Chaabouni, Garima Chadha, Jetha Chan, Blake Jianhang Chen, Jesse Chen, Lin Chen, Xu Chen, Derek Cheng, Tzu-hsiang Chien, Nikolai Chinaev, Yi Chou, Zhaohui Chu, Benjamin Coleman, Pooja Consul, Sam Conway-Rahman, Scott Crowell, Dylan Cutler, Vivek Dani, Samira Daruki, Anil Das, Daniel Deutsch, Nishanth Dikkala, Li Ding, Qiuhan Ding, Shenil Dodhia, Konstantin Donhauser, Tulsee Doshi, Anca Dragan, Alex Druinsky, Sahil Dua, Zoltan Egyed, Danielle Eisenbud, Daniel Eppens, Cindy Fan, Bahare Fatemi, Yassir Fathullah, Vlad Feinberg, Milen Ferev, Sebastian Flennerhag, Takumi Fujimoto, João Gabriel Oliveira, Isaac Galatzer-Levy, João Gante, Simon Geisler, Soham Ghosal, Antonious M. Girgis, Tamara von Glehn, Alec Go, Alhaad Gokhale, Alex Grills, Yiming Gu, Mayank Gupta, Pramod Gupta, Guru Guruganesh, Raia Hadsell, Hamza Harkous, Jitendra Harlalka, Demis Hassabis, Anja Hauth, Joe Heyward, Arian Hosseini, Chih-Yang Hsia, I-Hung Hsu, Xiaopeng Huang, Yangsibo Huang, Kevin Hui, Adrian Hutter, Te I, Fotis Iliopoulos, Advait Jain, Ganesh Jawahar, Ziwei Ji, Qilin Jin, Melvin Johnson, Kandarp Joshi, Arun Kandoor, Wang-Cheng Kang, Koray Kavukcuoglu, Mehran Kazemi, Kathleen Kenealy, Amr Khalifa, Phoebe Kirk, Ivan Korotkov, Suraj Kothawade, Vitaly Kovalev, Neel Kovelamudi, Adam Kraft, Ravin Kumar, Vivek Kumar, Harish Kuppam, Justin Lannin, Chen-Yu Lee, Seungji Lee, Dmitry Lepikhin, Alon Levkovitch, Dongdong Li, Qiujia Li, Valentin Liévin, Ethan Lin, Ziqian Lin, Casper Liu, Tianlin Liu, Tianqi Liu, Xin Liu, Ivan Lobov, Mayank Lunayach, Min Ma, Gagan Madan, Andrii Maksai, Eric Malmi, Michal Matuszak, Daniel McDuff, Gaurav Menghani, Maciej Mikuła, Daniil Mirylenka, Karolis Misiunas, Vedant Misra, Andreea Mitran, Kareem Mohamed, Maksim Mukha, Eric Noland, James O'Donnell, Brendan O'Donoghue, Kate Olszewska, Bernett Orlando, Wanqiong Pan, Rina Panigrahy, Unnati Parekh, Nicolas Perez-Nieves, Chunjong Park, Eric Paskie, Liqian Peng, Bryce Petrini, Slav Petrov, Jonas Pfeiffer, Bilal Piot, Martyna Plomecka, Siim Poder, Octavio Ponce, Arijit Pramanik, David Racz, Anish Rajan, Michelle Ramanovich, Anand Rao, Marvin Ritter, Vitor Rodrigues, Evan Rosen, Mikołaj Rybiński, Noveen Sachdeva, Michaël E. Sander, Rohit Sathyanarayana, Sagar Savla, Samuel Schmidgall, Tal Schuster, George Scrivener, Benoit Seguin, Andrew Sellergren, Aliaksei Severyn, Izhak Shafran, Dhruv Shah, Bobak Shahriari, Yuan Shangguan, Ashish Shenoy, Pradeep Shenoy, Rakesh Shivanna, Pauline Sho, Lucas Spangher, Wojciech Stokowiec, Tim Strother, Yao Su, Yinghao Sun, Mukund Sundararajan, Andrea Tacchetti, Mor Hazan Taege, Pouya Tafti, Jean Tarbouriech, Chetan Tekur, Shantanu Thakoor, Rahul Thapa, Madeleine Traverse, Lenart Treven, Tao Tu, Chien Te Tung, Çağlar Ünlü, Petar Veličković, Malini Pooni Venkat, Sagar Gubbi Venkatesh, Vidya Venkiteswaran, Francesco Visin, Alex Vitvitskyi, Kiran Vodrahalli, Weiyi Wang, Xin Wang, Tris Warkentin, Jan Wassenberg, John Wieting, Cindy Wu, Lechao Xiao, Hao Xu, Yuhui Xu, Fuzhao Xue, Arun Yadav, Jun Yan, Antoine Yang, Lin Yang, Ming-Hsuan Yang, Ziyu Ying, Jae Hyeon Yoo, Morteza Zadimoghaddam, Sajjad Zafar, Fred Zhang, Jiageng Zhang, Jianyi Zhang, Xiaofan Zhang, Chao Zhao, David Zhou, Chen ZouSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-02770.git
Cite
Version link
https://openversions.com/index/arxiv-2607-02770/commit/9610ff9a46c43aa086cb82ff8c0a86bfc9f95183
Version identifier (commit hash)
9610ff9a46c43aa086cb82ff8c0a86bfc9f95183

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2607.02770v2 OpenVersions: c852f436-8cd1-4ca6-aca7-57e82169105e

Posted

Claims

Findings

Headline finding

Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.

Encoding position with pp-RoPE (p=0.25p = 0.25) on global attention layers and RoPE on local attention layers, combined with KV cache sharing and reusing keys as values in global layers, reduces the global KV cache footprint by up to 37.5%.

Applying QAT (W8A8) to the 150M image encoder yields a 2x reduction in total forward-pass memory footprint (from 400 MB to 200 MB) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware, while QAT on the audio encoder achieves a 78% reduction in on-disk footprint (from 390 MB in Gemma 3n to 87 MB).

Gemma 4 31B is the top open model in the dense category on Arena Text, and both Gemma 4 31B and 26B-A4B show performance equal to much larger open models.

Gemma 4 31B is significantly better than Gemma 3 27B across the board on static benchmarks, while Gemma 4 E2B roughly matches Gemma 3 27B performance with 10x fewer parameters.

Compared to Gemma 3n of corresponding sizes, Gemma 4 achieves a 12% (E2B) and 10% (E4B) relative improvement on audio translation and a 17% (E2B) and 12% (E4B) relative improvement on audio transcription, despite a 78% reduction in on-disk audio encoder footprint.

The encoder-free Gemma 4 12B model achieves competitive audio-text performance on supported languages without a dedicated audio encoder.

Gemma 4 models significantly outperform Gemma 3 and 3n models in improving safety, while keeping unjustified refusals low.

Methods

Gemma 4 uses a decoder-only Transformer architecture with pre-norm and post-norm RMSNorm and QKNorm, comprising dense models (E2B, E4B, 12B, 31B) and a Mixture-of-Experts model (26B-A4B with 3.8B active parameters), where E2B and E4B use per-layer embeddings.

Source excerpts

“Gemma 4 models follow a decoder-only Transformer architecture (Vaswani et al., 2017). Our models have pre-norm and post-norm with RMSNorm (Zhang and Sennrich, 2019), and QKNorm (Henry et al., 2020).” · Show in paper

“The Gemma 4 family of models comprises dense architectures, with effective 2.3B (E2B), effective 4.5B (E4B), 12B and 31B parameters, as well as an MoE model with 3.8B activated parameters for 26B total parameters (26B-A4B). E2B and E4B use per-layer embeddings as in Gemma 3n (Gemma Team, 2025b), making them 2.3B and 4.5B effective out of 5B and 8B total parameters respectively.” · Show in paper

Long-context efficiency is achieved using 4:1 (E2B) and 5:1 (other models) local-to-global attention block ratios, reusing keys as values in global attention layers (except E2B and E4B), pp-RoPE (p=0.25p=0.25, 1M frequency) on global layers, RoPE (10k frequency) on local layers, and KV cache sharing (20/35 for E2B, 18/42 for E4B).

Source excerpts

“Our local to global attention ratio patterns follow Gemma Team (2025a), that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values = keys. We encode position with p-RoPE with p = 0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.” · Show in paper

For vision and audio modalities, E2B/E4B/26B-A4B/31B use ViT vision encoders (150M or 550M) with patch size 16, axial 2D-RoPE, 2D absolute positional embeddings, and aspect-ratio preserving resizing, while E2B/E4B use a 305M USM-based Conformer audio encoder processing 40ms chunks with Mel filterbank inputs.

Source excerpts

“E2B and E4B Gemma models come with a 150M vision encoder, while larger models use a 550M encoder (except for the unified 12B). Both are Vision Transformers (Dosovitskiy et al., 2021, ViT) with a patch size of 16, whose architectural differences are detailed in Table 10 in Appendix. Our vision encoders support variable aspect ratios (see Figure 2 and Algorithm 1) and incorporate both axial 2D-RoPE (Heo et al., 2024) with non-causal attention and 2D absolute positional embeddings.” · Show in paper

“E2B and E4B Gemma models use a 305M audio encoder that processes audio in 40ms chunks with Mel filterbank inputs. The encoder architecture is based on the Universal Speech Model (Zhang et al., 2023, USM), consisting of two downsampling convolution layers followed by twelve Conformer layers (Gulati et al., 2020).” · Show in paper

Gemma 4 12B uses an encoder-free architecture that replaces separate vision and audio encoders with a 35M-parameter matmul for 48x48x3 RGB patches (with 2D coordinate-based positional embeddings and LayerNorm) and direct projection of 40ms 16kHz raw audio chunks (640-dimensional vectors) into the LLM embedding space.

Source excerpts

“For the vision modality, Gemma 4 12B takes in 48x48x3 RGB patches, but replaces the 550M vision encoder by a single large matmul (35M parameters). Spatial awareness is maintained by adding 2D coordinate-based positional embeddings directly to the patch representations before a final LayerNorm layer (Ba et al., 2016).” · Show in paper

“For audio, the 305M USM-based conformer encoder is entirely discarded. Raw audio is segmented into 40ms chunks at 16kHz, resulting in 640-dimensional vectors per chunk. These are projected directly into the LLM embedding space.” · Show in paper

Quantization-Aware Training (QAT) is applied to produce mobile-quantized (per-channel int2/int4 weights, int8 activations) and Q4_0-quantized checkpoints with per-block scalar scales for fp16 inference, as well as W8A8 image encoders and mixed {2, 4, 8}-bit weight / 8-bit activation audio encoders.

Source excerpts

“we focus on two sets of weight representations: mobile quantization: per-channel low bitwidth weight (mix of int2 and int4) and activation quantization (int8). Q4_0 quantization: blockwise quantization, often referred to as Q4_0.” · Show in paper

“Furthermore, to enable stable inference in fp16, we introduce a scalar scale at each block in order to bound the activation ranges to fit fp16.” · Show in paper

An autoregressive Multi-Token Prediction (MTP) drafter head is trained for speculative decoding, using the main model's last-layer activations and token embeddings fed into a separate embedder and a 4-layer Transformer block that cross-attends to the main model's KVs.

Source excerpts

“In our MTP procedure, the model's last layer activations from the previous step and token embeddings are fed into the MTP head. The MTP head generates future tokens sequentially using a separate embedder and a 4-layer Transformer block that cross-attends to the KVs of the main model (Figure 1), thus eliminating the need for MTP prefill and supporting any draft length.” · Show in paper