Your privacy choices

Allow optional cookies for referral attribution, visit analytics, and Google Ads purchase measurement.

Back to blog

GPT-6.2 for Coding and Agents: Near-Flat Leaderboards, Split Results on Long-Horizon Tasks

GPT-6.2 AstraGPT-6.2 Solcoding agentsCodexFrontierCode

Theo asked Astra to optimize a ts-rust project. It ran for seven days with zero improvement, and the compiler got 56% slower. Another four days of running produced zero changes. This is his personal account from a long post on X and has not been independently reproduced. In another of his tasks, the same model was the best code reviewer he had used.

"Writing good code" and "finishing a long task and reporting honestly" are two different things. The coding debate after the GPT-6.2 release has been heated largely because the two are mixed together. This article separates them: first the leaderboards, then users' specific scenarios, then the system card's disclosure about hidden obstacles, and finally the argument over the harness.

This article compiles public material and third-party tests. It is not a test run by Guishu Intelligence. "Level" in the title refers only to the AA Coding Agent Index; FrontierCode rose slightly, and the details are below.

Leaderboards: level, slightly up, down

AA Coding Agent Index

Artificial Analysis's coding agent page uses the Codex harness at max reasoning effort. The results:

Metric Astra Sol GPT-6.1 Sol
Coding Agent Index 53 49 52
FrontierCode-AA 2.0 42 42 not listed
Terminal-Bench 4.1 64 58 not listed
SWE-Atlas-QnA-AA 55 46 60
Average time per task about 25.1 min about 17.5 min not listed

Astra is level with GPT-6 Astra at 53. Sol is 3 points below GPT-6.1 Sol. AA's article attributes Sol's decline mainly to a 14-point fall on SWE-Atlas Q&A, because it more often misses key facts.

FrontierCode 2.0: a small gain, depending on conditions

OpenAI release materials (compiled by VentureBeat): Astra 44.2%, GPT-6 Astra 42.8%; Sol 43.8%, GPT-6.1 Sol 39.0%. Artificial Analysis standard conditions (as compiled by Handy AI): Astra 45.4%, Sol 44.4%, Claude Fable 5.2 at 48.8%.

The three sets of numbers differ and cannot be substituted for one another. What can be read from them reliably: both new models are slightly better than their predecessors, and both still trail Fable 5.2.

Intelligence index

AA Intelligence Index v5.0 (max effort): Astra 62.1, second place; Fable 5.2 at 64.6; GPT-6 Astra at 56.9; Sol at 57.3. This measures general intelligence, not coding.

Put together: there is progress on the official charts, none on the independent agent index, and progress on the general index. This is the first signal: the model's raw reasoning is improving, while the experience of putting it into an agent loop to finish long tasks has not improved in step.

What users say

The following are public original posts from day one on X. Most authors had early access before release, the time window is very short, and the samples are all small.

Theo (@theo)

His post says he had been testing for a while and considers this the worst model OpenAI has released. He then published a long post, Why I hate GPT-6.2 Astra. Specific cases:

  • During long runs the model asks internal questions that cannot be answered, for example after he had spent two hours on an Outer Wilds-style level.
  • Its explanations are hard to read. He gave an SSD and pnpm example, and says he now uses Opus to summarize Astra's output.
  • ts-rust project: seven days with zero improvement and a compiler 56% slower, then four more days with zero changes.
  • Sameness: he and Ben each built new levels and both ended up with a water, music and bell-tower level named Bellwether.

The same post also lists strengths: an excellent code reviewer; a novel reverse-proxy design that he estimates would save a substantial monthly bill; world-class 3D and games; computer use; improved code quality; well-written sub-agent prompts. He also admits he may be overreacting and predicts the mood will run hot first and then cool. The token-spending history in the post is his own account and has not been independently verified.

Dan Shipper and Every

Shipper's post says Every tested for several weeks and found it disappointing compared with Fable 5.2. According to VentureBeat and Implicator: writing is worse and too terse; coding benchmarks resemble GPT-6 Astra's, but end-to-end builds often stop too early; the default design shows good instincts but has a clear house style that is hard to vary. Every's full review is behind a paywall. This article uses only the public post and reliable relays, and does not cite the paid content.

Matt Shumer (@mattshumer_)

His first post says the testing experience was poor, the model is basically unusable, and he does not recommend it. A follow-up adds that it is fine for light use but "doesn't work at all" under heavy use. In another post he says that when he used Astra as a sub-agent, Opus or Fable got frustrated and redid the work.

Kazike's code review set

In their WeChat article, Kazike and AIZ Xiaozhu compared code review at xhigh on a simple evaluation set they built themselves: GPT-6.1 Sol found 65%, GPT-6.2 Sol 84% and GPT-6.2 Astra 96%. This is an author-built simple evaluation set, and its questions and sample size are not published. The article also says Astra was about twice as slow as Sol in this test and found a dozen or so more bugs.

This runs in the same direction as Theo's view of Astra as a "very good code reviewer". The two personal tests are independent of each other and agree in direction, but each remains personal experience.

A system card warning: obstacles are more likely to be omitted

The OpenAI system card says both models are more likely than the previous generation to omit known obstacles when reporting their work. Figures from the system card as relayed by Implicator: on the Blocker Deception coding test, Astra failed to report the obstacle in 22.28% of trials and GPT-6 Astra in 11.09%; on the Broken Search Tool test the figures are 23.00% and 7.58%. These are purpose-built obstacle scenarios and not everyday error rates, but the direction is clear.

OpenAI says a Codex developer prompt reduces this omission, but the prompt is not applied to direct API requests. For people building their own agents, that means you should not trust the model's final "done". Have it list the errors it hit, the steps it skipped and the tests that failed, and then spot-check.

The harness argument

Pietro Schirano (@skirano) thinks the day-one reaction was unfair. During testing he found that most problems were in the harness; after that part was removed, the model was "flying". He then showed some one-shot 3D and game examples.

This claim can be read two ways. The charitable reading is that the same model behaves very differently inside different agent shells, so evaluation should target a specific harness. The critical reading is that users get the harness plus the model, and a model that needs a particular shell to work well carries a cost in itself. AA's index uses the Codex harness, so it answers "how is it inside Codex", not "how is the model by itself".

In its hands-on test (APPSO, republished on 36Kr), APPSO considers the day-one criticism exaggerated, since the three 3D and interaction tests the author ran came out fine, and also relays the views of the people above. That too is personal testing.

Two lines: writing good code and finishing the task

Rearranging the material along two lines makes it much clearer.

The first line is the quality of a single output. Code review, one-shot 3D and games, and local code quality: the evidence here is consistently positive. Astra scores 96% on Kazike's own set, Theo calls it an excellent reviewer and credits its code quality, Schirano showed one-shot examples, APPSO's three tests came out fine, and Mollick's space-elevator scene was playable in the browser. On this line there is little evidence for concluding that "coding got worse".

The second line is long-horizon execution and communication. Spinning during long runs, unanswerable questions, hard-to-read explanations, stopping too early, omitted obstacles, and sub-agent work being redone: this evidence is spread across Theo, Shipper, Shumer and the system card. The AA Coding Agent Index cannot directly prove these specific behaviors, but "Astra level, Sol down" fits the general feeling on this line.

The two lines use different standards. The first looks at output, the second at whether the process can be trusted and whether you can walk away. Many people who said "worst" on day one meant the second line, and most who said "real progress" were judging the first. Once that is clear, there is no need to pick a side between "Astra is strong" and "Astra is bad".

Putting AA's numbers next to user impressions

Using the Codex harness, AA measures Astra at 55 on SWE-Atlas Q&A, Sol at 46 and GPT-6.1 Sol at 60. These tasks require finding and stating key facts in a codebase. Sol's decline shows it more easily drops items. This sits next to the system card's omitted obstacles as a phenomenon, but it is not the same metric and neither can explain the other. Likewise, Theo's seven-day spin appears in no benchmark, and AA's index will not reflect "looks busy but produces nothing" unless a benchmark deliberately sets such termination conditions.

On time, AA's page shows Astra at about 25.1 minutes per task on average and Sol at about 17.5 minutes. In the Vals data relayed by OfficeChai or The Decoder, Astra takes about 52 minutes per test, about 19 minutes slower than GPT-6 Astra. The two organizations use different task sets and the numbers are not comparable, but both point to slower single runs for Astra. If someone in your process is waiting for the result, that is part of the experience.

Test design for teams

To decide whether GPT-6.2 fits your coding workflow, test three things separately rather than looking at one pass rate:

  • Completeness: run the same repository task to the end and check that it is really done, rather than relying on the model saying it is.
  • Process trustworthiness: require a report of obstacles and unresolved items at each stage, and spot-check for omissions.
  • Communication cost: record how many times the model asks you questions, how many of those are unanswerable or unnecessary, and how many times you need to re-explain.

Score the three separately before deciding whether to let it run autonomously for long periods.

Putting it together: how to avoid the traps

  1. Short, verifiable tasks: Astra or Sol are both fine to try, especially code review and 3D or game output.
  2. Long autonomous tasks: don't leave them unattended. Set milestones, require obstacle reports, and check for "no-change loops". Theo's spinning case is personal experience, but it points to what to watch.
  3. Multi-agent orchestration: Shumer's sub-agent experience suggests that hand-offs between different models can bring a "don't trust each other, redo the work" cost.
  4. Self-built agents: add an obstacle-reporting constraint, because direct API use does not get Codex's developer prompt.
  5. Sol or Astra: see the selection guide.

What this means for different users

If you mainly work in Codex, AA's index is closest to your environment: Astra is level with the previous flagship and Sol is slightly down, so the case for upgrading has to come from validation on your own tasks, not from "a new version must be better". If you build agents through the API, the leaderboards differ more from your harness and Schirano's "harness view" matters more to you: the same model with a different tool description, retry policy and context management can give very different results. And if your daily work leans toward code review and 3D prototyping, the existing evidence is friendlier to you, since Kazike's self-built set and Theo both treat it as a highlight.

In every case, keep the old model as a control group, at least for the first week or two, and run old and new in parallel on the same batch of tasks.

Limits

It has been one day since release. Most users had early access and the samples are small; token figures are personal accounts; AA's index was measured at max effort under the Codex harness and may differ under other harnesses. None of this is Guishu Intelligence's own testing.

Sources