OpenAI says the same model moved from 13.3% to 38.3% on ARC-AGI-3 when the evaluation harness retained reasoning and compacted context between actions. The post describes this as a change in how the system is run, not a newly trained model. It reports roughly six times fewer output tokens while retaining the model's reasoning state, and places the result beside a human baseline of 48%.
The harness changed the result
The comparison matters because an agent can lose useful state before the task is finished. OpenAI describes a rolling truncation strategy that discarded earlier reasoning after about 175,000 characters. The newer Responses API path keeps response IDs and uses compaction so the model can continue with a shorter context while preserving the reasoning state that the harness needs.
The reported numbers are therefore a package result: model, prompt, tool loop, context policy, compaction behavior and scoring. Calling 38.3% a model-only capability would erase the variable the post is trying to highlight.
13.3% versus 38.3%
| Reported setup | ARC-AGI-3 score | Context behavior |
|---|---|---|
| Older harness | 13.3% | Earlier reasoning discarded after rolling truncation |
| Retained reasoning and compaction | 38.3% | Response state retained with compaction between actions |
| Human baseline reported by OpenAI | 48% | Not a model run |
OpenAI also reports that the retained-reasoning setup used about one-sixth as many output tokens. That is a vendor-reported efficiency comparison under the described harness, not a guarantee for another task, model or context window.
Why benchmark settings matter
ARC-AGI-3 is designed to expose a model's ability to adapt inside interactive environments. The result can still be sensitive to tool timing, state retention, context limits, action serialization and scoring. The report does not establish an independent rerun or a general ranking across providers.
Re-run before choosing a model
- Pin the model, prompt, tool protocol and benchmark revision.
- Run the older and retained-reasoning harnesses on the same tasks.
- Record score, output tokens, compaction events, failures, wall time and cost.
- Repeat on a private interactive task that resembles your own workflow.
- Keep model quality, harness quality and operational cost as separate fields.
Watch for independent ARC-AGI-3 reruns and transparent task-level results. The immediate takeaway is practical: before replacing a model, check whether your agent is throwing away the context it needs to finish the job.
Sources: OpenAI — ARC-AGI-3, OpenAI conversation state and OpenAI compaction guidance.