By OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclerc, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang, Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojciech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, Shengjia ZhaoSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2508-10925.git
Cite
Version link
https://openversions.com/index/arxiv-2508-10925/commit/21edf475446fe4b455cb5d76938a121febc8734d
Version identifier (commit hash)
21edf475446fe4b455cb5d76938a121febc8734d

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2508.10925v1 OpenVersions: 1a96af78-ad15-4cc6-9c1b-b6f9c2964e10

Posted

Claims

Findings

Headline finding

gpt-oss-120b surpasses OpenAI o3-mini and approaches OpenAI o4-mini accuracy on canonical reasoning, coding, and tool-use benchmarks at reasoning level high, while gpt-oss-20b is competitive despite being 6 times smaller.

The gpt-oss models demonstrate smooth test-time scaling of accuracy when increasing the reasoning level (low, medium, high), generally showing log-linear returns where longer chains of thought provide higher accuracy at increased latency and cost.

On HealthBench and HealthBench Hard, gpt-oss-120b at reasoning level high nearly matches OpenAI o3 performance and outperforms GPT-4o, OpenAI o1, OpenAI o3-mini, and OpenAI o4-mini by significant margins.

On default safety evaluations, gpt-oss-120b and gpt-oss-20b perform on par with OpenAI o4-mini on disallowed content, jailbreaks (StrongReject), and fairness/bias (BBQ), while underperforming o4-mini on instruction hierarchy and hallucination (SimpleQA, PersonQA) evaluations.

Neither the default gpt-oss-120b model nor its adversarially fine-tuned versions reach OpenAI's Preparedness Framework indicative thresholds for High capability in Biological and Chemical risk, Cyber risk, or AI Self-Improvement.

Releasing gpt-oss-120b does not significantly advance the frontier of hazardous biological capabilities in open-weight foundation models, as existing open-source models such as Qwen 3 Thinking and Kimi K2 already score at or near adversarially fine-tuned gpt-oss-120b on most biosecurity-relevant evaluations.

Methods

Post-training quantization of Mixture-of-Experts (MoE) weights to MXFP4 format (4.25 bits per parameter) to reduce memory footprint, enabling gpt-oss-120b to fit on a single 80GB GPU and gpt-oss-20b to run on systems with 16GB memory.

Source excerpts

“We post-trained the models with quantization of the MoE weights to MXFP4 format[5], where weights are quantized to 4.25 bits per parameter. The MoE weights are responsible for 90+% of the total parameter count, and quantizing these to MXFP4 enables the larger model to fit on a single 80GB GPU and the smaller model to run on systems with as little as 16GB memory.” · Show in paper

Autoregressive Mixture-of-Experts (MoE) transformer architecture with Pre-LN RMSNorm, top-4 expert routing (128 experts for 120b, 32 for 20b), gated SwiGLU activations, alternating banded window (128-token bandwidth) and dense attention with Grouped Query Attention (64 query heads, 8 KV heads), RoPE, YaRN context extension to 131,072 tokens, and a learned bias in the softmax denominator.

Source excerpts

“Each MoE block consists of a fixed number of experts (128 for gpt-oss-120b and 32 for gpt-oss-20b), as well as a standard linear router projection which maps residual activations to scores for each expert. For both models, we select the top-4 experts for each token given by the router, and weight the output of each expert by the softmax of the router projection over only the selected experts.” · Show in paper

“Following GPT-3, attention blocks alternate between banded window and fully dense patterns [10][11], where the bandwidth is 128 tokens. Each layer has 64 query heads of dimension 64, and uses Grouped Query Attention (GQA [12][13]) with 8 key-value heads. We apply rotary position embeddings [14] and extend the context length of dense layers to 131,072 tokens using YaRN [15].” · Show in paper

Post-training with Chain-of-Thought (CoT) reinforcement learning using the custom harmony chat format, which enforces a role-based instruction hierarchy (System > Developer > User > Assistant > Tool), visibility channels (analysis, commentary, final), variable reasoning effort levels (low, medium, high), and agentic tool use (browsing, Python, developer functions).

Source excerpts

“For the models' training, we use a custom chat format known as the harmony chat format. This format provides special tokens to delineate message boundaries and uses keyword arguments (e.g., User and Assistant) to indicate message authors and recipients.” · Show in paper

“We train the models to support three reasoning levels: low, medium, and high. These levels are configured in the system prompt by inserting keywords such as "Reasoning: low".” · Show in paper

Safety training via CBRN pre-training data filtering and post-training deliberative alignment and instruction hierarchy supervision, while avoiding direct optimization pressure on the Chain-of-Thought (CoT) to preserve CoT monitorability.

Source excerpts

“During post-training, we use deliberative alignment[29] to teach the models to refuse on a wide range of content (e.g., illicit advice), be robust to jailbreaks, and adhere to the instruction hierarchy[30].” · Show in paper

“In accord with these concerns, we decided not to put any direct optimization pressure on the CoT for either of our two open-weight models.” · Show in paper

Adversarial fine-tuning methodology for Preparedness Framework risk assessment, combining helpful-only RL training to reward compliance with unsafe prompts and incremental domain-specific RL (end-to-end web browsing and biorisk human expert data for biology; CTF environments for cyber) at maximum reasoning effort.

Source excerpts

“We focus on incremental reinforcement learning, which we believe is the most apt technical approach. We use our internal OpenAI o-series RL training stack, which adds new capabilities while preserving the model's reasoning behavior. During training and evaluation time, we use the highest reasoning setting on gpt-oss.” · Show in paper