OpenAI's harness settings lift ARC-AGI-3 score to 38.3%
OpenAI says GPT-5.6 Sol jumps from 7.8% to 38.3% on ARC-AGI-3 when run through its Responses API with retained reasoning and context compaction.
Same model, different plumbing
OpenAI published a set of experiments arguing that GPT-5.6 Sol's weak showing on ARC-AGI-3 was largely an artifact of how the benchmark was run. In the standardized public harness the model scored 7.8%, because reasoning traces were discarded after each move and context was truncated rather than summarized. Re-running the same benchmark through OpenAI's Responses API with two switches enabled — retained reasoning between steps, and context compaction instead of truncation — lifted the score to 38.3%, above the 30.2% recorded for Anthropic's Claude Opus 5 earlier this week. OpenAI reported a 188% relative gain on the public set and framed the takeaway broadly: a benchmark number measures a model plus a bundle of largely invisible harness and API choices.
ARC Prize draws a line
ARC Prize co-founder François Chollet responded by distinguishing two kinds of modification. A harness purpose-built to beat the benchmark breaks the rules; general-purpose API settings that any developer can turn on do not, provided the settings and the resulting cost are disclosed. He conceded the arrangement creates parity problems between providers, but treated transparent reporting as an acceptable remedy. The official leaderboard deliberately avoids vendor-specific features so that every entry runs the same way — a design choice that, on OpenAI's account, penalized its model for capabilities the standard harness never exercised.
Why it matters
ARC-AGI-3 tests whether a model can learn unfamiliar 2D games with no instructions, exactly the kind of long-horizon, many-step task where carrying reasoning forward compounds. A fivefold swing from configuration alone is therefore not a footnote; it means published agentic scores are only comparable when harness, memory policy and token spend are specified alongside them. Enterprises picking between frontier models on leaderboard deltas are, in practice, comparing integration decisions as much as model quality. Expect pressure on benchmark maintainers to report multiple configurations per model, and on labs to state exactly which API features produced a headline number.