OpenAI launched GPT-6 Astra in September 2026 with a headline number that traveled fast: 99.9% on ARC-AGI-3, a benchmark built to resist memorization and reward genuine reasoning. Nvidia CEO Jensen Huang cited the figure days later when he declared that AGI had arrived. Then the people who run the benchmark checked the work. On ARC Prize’s standard, provider-neutral harness — the same setup every other lab’s model must clear — Astra scored 62.7%. That is still the best score on the benchmark, and still a real leap. But it is not 99.9%, and the 37-point gap between the two numbers is the most useful thing this launch taught anyone who buys, builds with, or writes about AI models.
This article is not a verdict on whether Astra is good. It is a guide to reading benchmark numbers without getting fooled, built from what actually happened in this case: two harnesses, one model, and a set of scores that only make sense once you know which setup produced each one.
What actually happened, in order
ARC-AGI-3 is not a multiple-choice exam. It drops an agent into unfamiliar, turn-based 2D environments — small games whose rules are never explained — and asks it to figure out the goal, try actions, observe what happens, and adjust. Humans clear these environments 100% of the time. When the benchmark launched in March 2026, frontier models looked lost next to that bar: Gemini 3.1 Pro managed 0.37%, GPT-5.4 scored 0.26%, Claude Opus 4.6 came in at 0.25%, and Grok 4.2 scored zero.
Astra changed that picture. OpenAI’s launch materials reported 99.9% on ARC-AGI-3, and the number spread through launch coverage and social posts as if it were the model’s score, full stop. ARC Prize then ran Astra itself, twice, and published both results. On its standard harness, Astra scored 62.7% at maximum reasoning effort. On OpenAI’s provider adapter harness, the same model scored 98.6% at maximum effort and 99.9% at high effort. The foundation’s verdict was blunt: the adapter score is not a comparable result, because it was not produced under the conditions every other lab’s model has to clear.
I want to be fair to both sides here, because the honest version of this story has two true halves. Astra’s neutral-harness result is genuinely strong — roughly double the next model on the same setup. And the harness moved the score by more than the entire gap between first and second place, which means the scaffolding has to be stated every time the number is quoted. Both halves matter, and dropping either one gives you a misleading story.
The two harnesses, side by side
A benchmark score is shorthand, and the shorthand hides the setup. Here is what actually differed between the two runs:
| ARC Prize standard harness | OpenAI provider adapter harness | |
|---|---|---|
| What the model keeps between requests | Only the notes it chooses to preserve | Its opaque reasoning state, carried across requests |
| Long conversations | No compaction; each request starts cold | Compaction reuses prior work instead of solving from scratch |
| Astra score at max effort | 62.7% | 98.6% |
| Astra score at high effort | 54.8% | 99.9% |
| Cost of the run | About $26,098 | About $18,817 |
| Comparable to other labs’ scores? | Yes — this is the shared setup | No — this setup is specific to one provider |
Two details in that table deserve a second look. First, the adapter run was cheaper, not more expensive: roughly $7,281 less than the standard run, even while scoring far higher. That is exactly what you would expect if preserved state removes redundant re-reasoning — the model does less repeated work. Second, on the 167 game-and-reasoning pairs both harnesses solved, adapter runs were about 3.66 times faster and used 49% fewer tokens. The adapter is not worthless. It is a better system. It is just not a model-only measurement, and presenting a model-plus-scaffolding result as a model result is a different claim.
Same model, five scores
This is the part I wish every launch chart printed. Hold the model fixed — GPT-6 Astra, unchanged weights — and change only the harness and the effort level, and you get a spread of more than 80 points:
| Setup | Score |
|---|---|
| Standard harness, low effort | 17.5% |
| Standard harness, high effort | 54.8% |
| Standard harness, max effort | 62.7% |
| Provider adapter, max effort | 98.6% |
| Provider adapter, high effort | 99.9% |
Read that table before you quote any single number from it. The difference between low effort and max effort on the same neutral harness is 45 points. The difference between harnesses at the same effort level is 36 to 45 points. Effort level and scaffolding together matter more than most model upgrades do.
And here is the comparison that actually ranks models against each other, all on the standard harness:
| Model | ARC-AGI-3, standard harness |
|---|---|
| GPT-6 Astra | 62.7% |
| Claude Opus 5 | 30.2% |
| GPT-5.6 Sol | 7.8% |
On identical scaffolding, Astra leads the next model by more than 30 points. That is a real result, and it deserves to be reported as one. It is also, notably, the number nobody put in a headline.
Why the harness moves the score so much
ARC-AGI-3 is an interactive benchmark, and interactive benchmarks punish forgetfulness. A static test looks like input, model, output. An interactive test looks like this: observe the environment, decide what matters, preserve the right state, choose an action, read the result, and adjust. Every step where the agent loses track of what it already tried is a step where it has to start over.
ARC Prize singled out one behavior of Astra’s that OpenAI did not lead with: the model keeps running notes, which the foundation described as remarkable for their precision and information density. That is a mechanism observation from outside the vendor, and frankly it tells you more than either percentage. A model that writes good notes about what it has already tried will waste less effort repeating itself — and in a test where the rules are unknown and must be discovered by playing, wasted effort is the main failure mode.
The adapter goes further than good notes. By preserving reasoning state between requests and compacting long conversations, it lets the agent reuse prior work instead of reconstructing it. That is a legitimate engineering advantage in a real product. Your coding assistant should remember what it did five minutes ago. But when you are comparing models, that advantage belongs in a separate column from the model itself — otherwise you are comparing one lab’s model-plus-system against another lab’s model alone.
The bigger story: scaffolding is now the capability
Here is where the Astra dispute stops being a fact-check and starts being a strategy lesson. The same month, two independent results showed that the software wrapped around a model can matter more than the model choice itself — with one important caveat about which test set each result used.
Prime Intellect’s open-source Prime Agent harness, built around persistent Python sessions and sub-agents that act as function calls, took Claude Opus 5 — the same model that scores 30.2% on ARC Prize’s standard harness — to 95.5% on the benchmark’s public set. Nvidia’s AVO harness reportedly took the same model to 100% on the public set. A separate R-native harness called corteza reported 98.58% with Opus 5 and 100% with Astra, again on the public set.
Notice the distinction I just made, because it is the whole ballgame: those 95.5% and 100% figures are on the public set, the levels everyone can see and practice on. Astra’s 62.7% is on ARC Prize’s verified, semi-private leaderboard. Public-set scores and semi-private scores are not the same measurement, and anyone who puts them in one ranking without saying so is making the same category error OpenAI’s headline made. But even with that caveat, the direction is unmistakable: better scaffolding lifted the same weights from around 30% into the mid-90s on the same class of task.
That reframes what you are actually buying when you pick a model. You are not buying a score. You are buying a model plus whatever harness your tool wraps around it — the memory, the tool access, the retry logic, the way it recovers from mistakes. Two products built on the same model can behave like different models entirely. And a cheaper model inside a better harness can beat a flagship model inside a thin one, which is exactly what the cost data from this launch suggests: OpenAI’s own later charts put GPT-6.1 Sol roughly level with Astra on some coding tasks at a fraction of the cost per task.
How to read any AI benchmark: the 5-question checklist
Use this the next time a launch chart lands in your feed. If you cannot answer all five questions from the vendor’s own materials, treat the headline as marketing, not measurement.
1. Which harness produced this number?
Look for words like “standard harness,” “provider adapter,” “tools-enabled,” or “custom setup” in the footnotes. OpenAI’s own footnotes did disclose the adjusted harness — the caveat was there, in small print, while the headline traveled without it. If the harness is not named, the number is not comparable to anything.
2. Were the competitors run on the same harness?
A chart is only a ranking if every bar in it was measured the same way. In OpenAI’s launch table, the ARC-AGI-3 row showed Astra at 99.9% against GPT-5.6 Sol at 7.8% and Opus 5 at 30.2% — but those competitor figures are standard-harness numbers sitting next to an adapter number. That is not a comparison. It is two different tests sharing one table.
3. What effort level, and what did it cost?
Effort is a dial, not a fixed property. Astra’s standard-harness score moved from 17.5% at low effort to 62.7% at max. A vendor quoting only its best effort level is quoting its most expensive configuration. Ask what the run cost — the $26,098 standard run versus the $18,817 adapter run tells you something real about efficiency — and ask what the score is at the effort level you would actually pay for.
4. Public set or private set?
Scores on tasks the model could have seen, practiced on, or been tuned against are weaker evidence than scores on held-out tasks. When a harness result cites the public set, say so, and do not rank it against semi-private leaderboard numbers. This single distinction deflates most of the “100% on ARC” claims circulating since August.
5. What does the score measure — the model, or the system?
This is the question to ask before any purchase decision. If you are buying an API, you care about the model. If you are buying a product — a coding tool, an agent platform, a support bot — you care about the system, and the adapter-style number may actually be the more relevant one for you. The mistake is not using the system number. The mistake is using it to claim the model alone did something.
A practical model-picker framework
Forget the leaderboard for a minute and run this instead, using your own work:
Step 1: Define five real tasks. Not benchmark tasks — your tasks. A bug fix in your codebase, a research summary from your documents, a multi-step workflow your team actually runs. If a model cannot be tested on your work, its benchmark score is the only thing you have, and that should worry you.
Step 2: Run every candidate through the same harness. Use the same tool, the same prompts, the same retry rules. If one vendor’s demo uses a fancier harness than the other’s, you have recreated the Astra problem in your own office.
Step 3: Score cost per completed task, not cost per token. A model that charges less per token but needs three retries and twice the reasoning is not cheaper. Record tokens used, time taken, and whether a human had to intervene. The adapter data from this launch — 3.66 times faster, 49% fewer tokens on shared tasks — is a reminder that efficiency and score move together when the scaffolding is good.
Step 4: Check the failure mode, not just the success rate. Did the model fail by giving up early, by repeating itself, or by confidently doing the wrong thing? Scaffolding fixes the first two. Nothing in a harness fixes a model that is confidently wrong, and that is the failure you most need to know about.
Step 5: Re-test when the harness changes, not just when the model changes. A vendor update to its agent framework can move your results more than a model version bump. Treat harness release notes as seriously as model release notes.
What to ignore in launch charts
A few patterns should now trigger immediate skepticism. A score of 99% or 100% on any benchmark that was designed to be hard is a claim about the test setup until proven otherwise. A chart where one bar towers over the rest while the footnote mentions a “custom” or “adjusted” setup is two tests pretending to be one. A record announced with no cost figure attached is hiding the effort dial. And any claim that a single score proves “AGI has arrived” is a category error: one benchmark, however good, measures one kind of task, and ARC-AGI-3 itself is a narrow slice of what general intelligence would mean.
None of this makes Astra unimpressive. Going from near-zero scores in March to 62.7% on a neutral harness in seven months is, as ARC Prize itself reportedly put it, a genuine step forward worth celebrating. It just is not 99.9%, and it is not AGI. The difference between those sentences is the difference between a useful signal and a misleading one — and in a market where teams are choosing vendors, budgets, and architectures off these numbers, that difference is worth real money.
Frequently Asked Questions
What is ARC-AGI-3?
ARC-AGI-3 is a benchmark from the ARC Prize Foundation that tests reasoning in unfamiliar, interactive environments. Instead of answering questions, the agent plays small 2D games whose rules are never explained, and must discover the goal by experimenting. It was designed to resist memorization, since the tasks are novel by construction.
What did GPT-6 Astra actually score on ARC-AGI-3?
On ARC Prize’s standard, provider-neutral harness, Astra scored 62.7% at maximum reasoning effort, 54.8% at high effort, and 17.5% at low effort. On OpenAI’s provider adapter harness, the same model scored 98.6% at max effort and 99.9% at high effort. Only the standard-harness figures are comparable to other models’ scores.
Why did ARC Prize reject the 99.9% figure?
Because it was produced with OpenAI’s own adapter harness, which preserves reasoning state between requests and compacts long conversations — conditions other labs’ models do not get. ARC Prize’s position is that a score only counts as a comparable result when every model clears the same setup.
Is GPT-6 Astra still the best model on this benchmark?
Yes, on the comparable measurement. Its 62.7% standard-harness score leads Claude Opus 5 at 30.2% and GPT-5.6 Sol at 7.8% by a wide margin. The dispute is about the size of the lead, not who holds it.
What is a harness, in plain English?
The harness is the software wrapped around an AI model during a test or a task: how it remembers past steps, what tools it can use, how it retries after mistakes, and how it manages long conversations. The same model inside a better harness can score dramatically higher, which is why harness disclosure matters as much as the model name.
Should I switch models because of this story?
Not on this story alone. Test candidates on your own tasks, through the same tool, and compare cost per completed task rather than headline scores. If you are buying a finished product rather than a raw model, the system-level result — harness included — may be the number that actually matters for you.
Sources: ARC Prize’s independent rerun and the harness comparison are covered in Startup Fortune’s report and Everyday Data Science’s breakdown of the two scores. For a deeper look at how Prime Agent’s scaffolding works, the Prime Agent interview with Seth Karten walks through the design. Internal context: see also Gemini 4 Argon, GPT-6 Astra, and Claude flagship pricing compared and Claude Opus 5.5’s cheaper launch.