GPT-5.6 Sol and GPT-6 Astra in Codex: higher intelligence, lower autonomous completion and operational reliability

September 7 evidence update: Connected analysis of delivery outcomes, successor launches, Sol supervision burden, Astra's launch and DeepSWE. It links the original Intake issue #40560, preserves accepted work and inconclusive comparisons, and distinguishes observed temporal recurrence from the unproved resource-allocation hypothesis. Pinned evidence bundle.

Environment

• Codex Desktop on Windows 11
• ChatGPT Pro subscription
• Models used inside Codex: GPT-5.6 Sol and GPT-6 Astra
• Model used in the related ordinary ChatGPT examples: GPT-5.6 Sol
• Repeated observations during August and September 2026
• Codex workload: long-running coding, repository editing, tool-heavy execution, research, debugging, and delivery tasks
• Related report: #42080: completion claims without target evidence and missing circuit breakers

Scope clarification: all reports in this issue about non-delivery, premature stopping, false completion, supervision burden, scope drift, and unreliable execution come from work performed in Codex, where I used both GPT-5.6 Sol and later GPT-6 Astra. Astra was not available in my ordinary ChatGPT model picker at the time. Ordinary ChatGPT is included separately because GPT-5.6 showed related failures in reasoning discipline, context interpretation, fact verification, and correction recovery. Those ChatGPT examples are not being presented as programming work through ChatGPT.

The long public-evidence analysis that I later pasted while discussing this regression was generated by GPT-6 Astra in Codex, not by GPT-5.6. The operational reliability problem, however, had already been observed with GPT-5.6 Sol in Codex before Astra was used.

Summary

I am reporting a cross-model regression in operational task completion, instruction fidelity, and supervision cost inside Codex, together with related reliability failures in GPT-5.6 Sol inside ordinary ChatGPT. This is not a blanket claim that newer models are less intelligent.

Both Codex models can produce excellent work. GPT-6 Astra can be noticeably more capable in analysis and difficult reasoning. The problem is the reliability of converting that capability into a complete, accepted delivery.

In my Codex workflow, the observed trajectory is:

• reasoning quality and best-case output improve;
• peak results become more impressive;
• autonomous completion becomes less predictable;
• more tasks end with analysis, planning, partial implementation, or a premature completion claim instead of the requested end state;
• more user checking, correction, continuation prompts, and scope restoration are required;
• GPT-6 Astra often exhibits the same failure pattern more severely than GPT-5.6 Sol.

The practical pattern is:

Intelligence goes up while operational reliability goes down.

More concretely:

The model becomes better at thinking and producing exceptional outliers while becoming less dependable at doing exactly what was asked, preserving the actual specification, checking uncertain claims, and continuing until the work is complete.

The primary product metric should therefore not be only benchmark success or the quality of the best run. It should include:

Accepted, complete deliverables per unit of human supervision.

What happens in Codex

Across unrelated tasks, the recurring pattern is:

1. I give Codex a bounded task with a clear deliverable.
2. The model often understands and restates the goal correctly.
3. It may perform useful analysis, inspect relevant files, or implement part of the work.
4. It then stops early, shifts into more planning, expands the scope, follows an inferred objective, or treats partial progress as completion.
5. I must ask it to continue, verify, finish, restore a constraint, undo an invented restriction, or return to the explicit objective.
6. The model may acknowledge the correction correctly in prose without reliably binding that correction to the next action.
7. Even after multiple interventions, the result may remain incomplete, low quality, or not fully compliant with the request.

This is not based on one bad run. Some tasks are completed beautifully. The problem is that too many ordinary tasks require continuous monitoring and repeated correction, and some still do not reach an acceptable end state.

Observed failure classes in Codex include:

• planning or explaining instead of executing;
• ending before the requested artifact, code state, fix, or delivery exists;
• claiming completion from component-level or proxy evidence;
• implementing only the easiest or most convenient portion of the task;
• silently ignoring one or more explicit requirements;
• forgetting constraints, files, or decisions already established in the session;
• acknowledging a correction without applying it operationally;
• correcting one requirement while breaking another;
• repeatedly using tools without converging on the declared end state;
• requiring the user to act as supervisor, QA, state store, specification owner, and continuation trigger.

Model comparison inside Codex

The table below describes my workflow observations, not a universal benchmark claim.

Dimension	GPT-5.6 Sol in Codex	GPT-6 Astra in Codex
Peak reasoning quality	Very high	Often higher
Best-case output	Occasionally excellent	Occasionally even more impressive
Autonomous completion	Unreliable	In my recent workflow, often worse
Need for user supervision	High	Frequently higher
Premature stopping or non-delivery	Recurring	Still recurring
Loss of task constraints	Present	Present, sometimes through a more elaborate reinterpretation
Unrequested assumptions	Present	More frequent or more forcefully applied
Invented rules and restrictions	Present in some cases	More pronounced in recent sessions
Scope drift	Disruptive	More sophisticated and harder to unwind
Confidence in an inferred specification	High	Often stronger, even when the inference is wrong

GPT-5.6 Sol in Codex

With GPT-5.6 Sol, the recurring problem is not an absence of useful work. It is that useful work frequently does not become a complete delivery without active supervision.

The model can inspect the right files, identify the right direction, and make meaningful progress, but still:

• stop before finishing;
• omit part of the requested work;
• claim readiness without validating the target state;
• require repeated continue, check, fix, or finish prompts;
• produce something usable but not complete or not fully correct.

GPT-6 Astra in Codex

Astra exhibits the same operational reliability problem and, in my workflow, often a worse form of it.

Recent Astra-specific behavior includes:

• inventing facts or contextual details that were not provided;
• inferring a possible rule and silently treating it as an explicit instruction;
• creating restrictions, process requirements, or prohibitions that I never requested;
• replacing my stated objective with a model-inferred objective;
• optimizing for a specification the model created instead of the specification I gave;
• continuing to enforce self-created constraints after correction;
• generating a persuasive explanation for the deviation instead of delivering the requested result;
• expanding the task into adjacent research or redesign when the requested delivery was narrower;
• requiring repeated supervision and still failing to deliver fully.

This is not a complaint about legitimate safety policy, tool authorization, or repository instructions. It is a complaint about self-authored specification drift on benign development tasks.

The stronger model can make this failure more damaging. A weaker model may make an obvious mistake. A stronger model may construct a coherent, detailed rationale around a wrong premise and then execute confidently against that premise. The result is harder to detect, harder to unwind, and more expensive to supervise.

The long research and evidence summary that I pasted during the discussion of this issue was itself produced by GPT-6 Astra in Codex. That attribution matters, but it does not mean that all earlier operational failures came from Astra. I had already experienced the supervision and completion problem with GPT-5.6 Sol in Codex. Astra continued the pattern and often intensified it.

Related GPT-5.6 reliability failures in ordinary ChatGPT

The Codex evidence above concerns coding execution and delivery. Ordinary ChatGPT showed a separate but related failure pattern: uncertain inferences were promoted into authoritative facts instead of being checked.

Incident 1: fabricated product architecture and model mapping

When I said that I was already using GPT-6 Astra, the context was Codex. The GPT-5.6 assistant in ordinary ChatGPT then:

1. transported that fact into the ChatGPT product surface without evidence;
2. conflated separate concepts such as model identity, reasoning effort, model-picker options, product surface, plan, and rollout availability;
3. treated Pro as though it automatically meant Astra;
4. produced a structured table and configuration recommendations based on that invented mapping;
5. received a direct correction from me;
6. instead of stopping and verifying, improvised a second, more detailed taxonomy that was also inconsistent with the actual state of my account;
7. implied that Astra was available in my ChatGPT selector even though I explicitly stated that my ChatGPT showed only GPT-5.6 Sol.

The problem was not merely an imprecise label. It was a fabricated synthesis presented with confidence, followed by a second fabrication after correction.

The assistant later described its own failure accurately: it had filled a knowledge gap by inventing a product architecture, transported Codex context into ChatGPT, mixed model and reasoning axes, and presented the result as fact.

Incident 2: failure to verify Voice Live, Gmail, apps, and MCP capabilities

In another ordinary ChatGPT exchange, I asked whether Voice Live could invoke my connected Gmail app, MCP tools, or custom actions. The assistant:

• inferred general product behavior from the tools visible in one session;
• gave contradictory answers about whether voice and text exposed the same integrations;
• claimed that the connected Gmail app should work through voice without first checking current product documentation;
• changed its explanation several times as I challenged it;
• verified the official information only after repeated demands.

Again, the correct next step was available: verify a current and mutable product capability before making a confident claim. The model instead generated an explanation from incomplete local evidence and treated that explanation as authoritative.

Problems visible in ChatGPT

The related ChatGPT reliability failures include:

• inventing missing product facts instead of verifying them;
• misreading which product surface the user is referring to;
• transporting context from Codex into ordinary ChatGPT without justification;
• conflating model identity, reasoning effort, plan, product surface, and rollout status;
• converting a tentative inference into a definitive table, taxonomy, or recommendation;
• ignoring the user's direct observation of what is visible in their own account;
• reacting to correction by generating a more elaborate rationalization instead of checking;
• citing or invoking documentation only after the unsupported conclusion has already been formed;
• giving configuration advice based on a product structure that was never established;
• requiring the user to act as verifier, source of truth, and repeated correction loop.

Shared failure pattern across Codex and ChatGPT

The surfaces and tasks are different, but the common mechanism appears to be:

The model promotes an inference into an authority.

In Codex, the inference can become a task requirement, restriction, replacement specification, or premature completion claim.

In ordinary ChatGPT, the inference can become a product fact, model mapping, capability claim, or configuration recommendation.

This matters because greater reasoning ability can make the unsupported premise more coherent and persuasive. The result may sound more rigorous while remaining less grounded in the explicit request, actual product state, or available evidence.

Why this matters

For production work, peak capability is not enough.

A model that produces one exceptional result but requires close supervision on most tasks may be less useful than a slightly less capable model that completes ordinary work consistently.

A coding agent has failed operationally when it:

• reasons correctly but does not finish;
• writes most of the solution but leaves critical requirements undone;
• reaches a plausible intermediate state and calls it complete;
• consumes significant time and quota while the user repeatedly restores the original objective;
• produces a final answer that sounds finished without an accepted deliverable behind it.

A conversational assistant has also failed operationally when it:

• could verify a current claim but guesses instead;
• turns uncertain context into a confident product explanation;
• receives a direct correction and rationalizes instead of checking;
• gives recommendations based on facts it invented.

There is no useful production category called “almost deployed,” “almost fixed,” or “mostly delivered” when the task required a completed result. Likewise, a polished explanation is not reliable if its factual structure was fabricated.

Suggested reproduction method

Because this is a stochastic reliability problem, one run is not sufficient.

Codex evaluation

1. Select several bounded repository tasks with:
   • an explicit deliverable;
   • a small set of constraints;
   • a clear done-when criterion;
   • tools sufficient to implement and verify completion.
2. Run the same task class multiple times with GPT-5.6 Sol and GPT-6 Astra in Codex.
3. Require execution, not only a plan or analysis.
4. Record whether the model:
   • reaches the target without a continuation prompt;
   • preserves every explicit constraint;
   • distinguishes facts from assumptions;
   • invents additional requirements or restrictions;
   • replaces the explicit objective with an inferred one;
   • validates the actual target state before claiming completion;
   • completes all requested deliverables.
5. After one incorrect assumption or deviation, correct the model once.
6. Measure whether the correction changes the next relevant action, rather than only the next explanation.
7. Stop measurement only when the user accepts the result or abandons the attempt, not when the model ends a turn.

ChatGPT evaluation

1. Ask current questions involving model identity, reasoning settings, rollout state, connectors, tools, or product capabilities.
2. Include one account-specific observation supplied by the user.
3. Give the model access to authoritative documentation or web verification.
4. Record whether it:
   • keeps model, reasoning effort, plan, product surface, and rollout state separate;
   • labels uncertainty instead of fabricating a taxonomy;
   • verifies mutable claims before answering confidently;
   • respects the user's observed account state;
   • responds to a direct correction by checking evidence rather than constructing a new rationalization.
5. Repeat across multiple runs and model versions.

Metrics that would capture the regression

I suggest adding supervision-aware, specification-fidelity, and evidence-grounding metrics to Codex and ChatGPT evaluations:

• Autonomous Completion Rate: percentage of tasks accepted without any user continuation or correction.
• Interventions per Accepted Deliverable: user messages required after the initial task definition.
• False Completion Rate: tasks declared complete without target-level evidence.
• Requirement Coverage at First Delivery: explicit requirements satisfied in the first submitted result.
• Premature Stop Rate: tasks ended while required work remained feasible and unfinished.
• Unauthorized Constraint Introduction Rate: restrictions or requirements added without provenance in user instructions, policy, repository rules, or tool limitations.
• Assumption-to-Rule Escalation Rate: uncertain inferences silently promoted into binding execution rules.
• Explicit-Intent Override Rate: model-inferred objectives displacing the stated user objective.
• Inference-to-Fact Promotion Rate: uncertain inferences presented as verified information.
• Product-Surface Confusion Rate: conflation of ChatGPT, Codex, API, plan, reasoning level, model identity, and rollout availability.
• Verification-Before-Confidence Rate: mutable claims checked against an authoritative source before being stated confidently.
• Correction Recovery Rate: probability that a direct correction triggers verification and changes the next answer or action rather than producing a new rationalization.
• Unsupported Configuration Advice Rate: recommendations based on an unverified or invented product structure.
• Correction Binding Rate: probability that a correction changes the next relevant action rather than producing only verbal agreement.
• State Refresh Compliance: whether relevant files, commands, external state, or documentation are reread after the user disputes a claim.
• Regression Introduction Rate: frequency with which fixing one requirement breaks another.
• Time and Tool Calls to Accepted State: measured at user acceptance, not turn termination.
• Quota per Accepted Deliverable: total usage divided by results the user can actually use.
• Peak-to-Median Reliability Gap: difference between the model's best demonstrations and its typical supervision-adjusted outcome.

The primary Codex metric should be:

Accepted, complete deliverables per hour of human supervision.

A second key Codex metric should be:

Probability that Codex completes the explicit user specification without inventing a replacement specification.

A key ChatGPT metric should be:

Probability that the assistant verifies a mutable factual claim instead of converting an inference into an authoritative answer.

Public evidence consistent with the pattern

The evidence does not establish a universal collapse, but several public reports and evaluations are consistent with localized or operational regressions:

• #41851 reports explicit constraints being acknowledged but not reliably bound to subsequent actions.
• #33816 reports GPT-5.6 losing a live command-session invariant, treating work as complete, and attempting duplicate execution.
• #32225 reports a bounded implementation task remaining incomplete after prolonged execution, with high orchestration and supervision overhead.
• A repeated 21-task evaluation of GPT-5.5 High found more time and tool calls with weaker results on several delivery-oriented measures, while some quality rubrics improved: https://www.stet.sh/blog/gpt-55-high-regression-check-graphql-go-tools
• A preprint on item-level migration regressions reports that aggregate model improvement can coexist with instruction-following regressions: https://arxiv.org/html/2608.17719v1

There is also contrary evidence. Some independent coding trackers report strong or nominal aggregate performance for newer models. OpenAI also reports substantial benchmark improvements for GPT-6 Astra. That does not invalidate this issue. Benchmark pass rate, peak reasoning, and supervision-adjusted delivery reliability measure different properties.

A benchmark often evaluates the final result after a bounded run. It does not necessarily measure:

• how often the user had to say continue;
• how many invented constraints had to be removed;
• how often a correction was acknowledged but not operationalized;
• whether the model stopped before completing the requested delivery;
• how much human attention and quota were required before acceptance;
• how often an assistant fabricated a coherent product explanation instead of verifying a mutable fact.

Expected behavior

Codex

1. Codex should not treat a task as complete until the declared deliverable or target state exists.
2. Turn completion must remain separate from task completion.
3. Explicit user instructions should take priority over model-inferred preferences or objectives, subject to actual policy, repository rules, and tool constraints.
4. Assumptions should remain labeled, uncertain, and reversible. They should not silently become binding rules.
5. The model should not create restrictions, prohibitions, or requirements without explicit provenance.
6. A user correction should refresh the relevant state and govern the next action.
7. Codex should preserve the active objective, constraints, rejected paths, and done-when criterion across long sessions and compaction.
8. Completion language should require target-level evidence.
9. Ordinary bounded tasks should not require repeated continue, finish, check, remove the rule you invented, or you still did not deliver prompts.

ChatGPT

1. Keep model identity, reasoning effort, plan, product surface, and rollout availability as separate variables.
2. Never infer the active model solely from a label such as Pro, High, or XHigh.
3. For current product capabilities and configuration, verify authoritative documentation before making a confident claim.
4. Treat the user's observed account state as account-specific evidence, not something to overwrite with a generic rollout assumption.
5. After a direct correction, stop rationalizing and verify before continuing.
6. Do not produce tables, taxonomies, or configuration recommendations until the underlying facts have been established.
7. Clearly label assumptions when authoritative verification is unavailable.

Requested remediation

• Evaluate GPT-5.6 Sol and GPT-6 Astra on identical repeated Codex tasks using user interventions and accepted completion as first-class outcomes.
• Add intervention-aware, accepted-deliverable, specification-fidelity, correction-recovery, and evidence-grounding metrics to internal evaluations.
• Preserve the active objective, explicit constraints, rejected paths, assumptions, done-when criterion, and readiness state across compaction.
• Track constraint provenance as user, policy, repository, tool, or model inference.
• Prevent model-inferred constraints from silently becoming hard requirements.
• Require target evidence before emitting completion language or writing a completed state through a tool.
• Treat user disagreement with a factual or operational claim as a trigger to verify or reread state, rather than reflexively agreeing or rationalizing.
• Add circuit breakers for repeated failures, unchanged evidence, excessive tool calls, and long execution without progress toward acceptance.
• Expose readiness states such as planned, implemented, validated-local, validated-target, released, and accepted.
• Evaluate ChatGPT on product-surface separation, verification before confidence, and recovery after direct correction.
• Provide a stable way to compare or pin model snapshots for production workflows.

What I am not claiming

• I am not claiming that every GPT-5.6 Sol or GPT-6 Astra task fails.
• I am not claiming that Astra is universally less capable than Sol.
• I am not claiming that every observed failure has one internal cause.
• I am not claiming that model behavior alone explains every Codex client, context, compaction, or orchestration failure.
• I am not claiming that the ChatGPT examples are coding-delivery sessions; they are separate evidence of reasoning, verification, context, and correction failures.
• I am not claiming that GPT-6 Astra was available in my ordinary ChatGPT account at the time. It was available to me in Codex.

I am claiming that peak intelligence is not a sufficient product-quality measure. In my Codex workflow, both GPT-5.6 Sol and GPT-6 Astra can produce exceptional results while still requiring too much checking, correction, continuation, and rework before a complete delivery exists, with Astra often making the reliability problem worse rather than better. In ordinary ChatGPT, GPT-5.6 has shown a related tendency to turn unsupported inferences into confident product facts and recommendations instead of verifying them.

The report names Codex Desktop on Windows 11 and links issues #40560 and #42080 plus an evidence bundle, but it names no repository files, tests, or implementation entry point. Start by reviewing those linked reports and the cited evidence; a concrete fix and acceptance criteria are not specified, so “done” remains undefined.