GPT-5.6: high peak capability, but lower autonomous completion and higher supervision burden

Environment

• ChatGPT Pro subscription
• Codex Desktop on Windows 11
• GPT-5.6 family, especially GPT-5.6 Sol, across long and tool-heavy tasks
• Repeated observations during August and September 2026
• Related existing report: #42080: completion claims without target evidence and missing circuit breakers

Summary

I am reporting a reliability regression in operational task completion, not a blanket claim that GPT-5.6 is less intelligent.

GPT-5.6 can produce exceptional results. The problem is that its median operational outcome appears less reliable: it often analyzes, replans, explains, or partially implements instead of delivering the requested end state. The user must repeatedly prompt it to continue, verify claims, restore constraints, correct scope, or finish work that the model has already described as complete.

The practical regression is therefore best expressed as:

Fewer accepted deliverables per unit of user supervision, despite high peak capability.

For production use, this distinction matters more than isolated benchmark gains. A model can reason better on individual steps and still be a worse agent if it requires constant human steering to reach the requested target.

What issue are you seeing?

Across unrelated tasks, the recurring pattern is:

1. The user gives a bounded task with a clear deliverable.
2. The model correctly restates the goal and constraints.
3. It performs useful intermediate work, but does not reach the declared end state.
4. It ends the turn, reports progress as completion, or shifts into additional analysis.
5. The user must ask it to continue, check, fix, verify, or return to the original objective.
6. The correction is acknowledged textually, but is not always bound to the next action.
7. The final result may remain incomplete, weak, or only partially compliant after several interventions.

Observed failure classes include:

• planning or explaining instead of executing;
• ending before the requested artifact or target state exists;
• claiming completion from component-level or proxy evidence;
• acknowledging constraints without reliably enforcing them in the next action;
• losing scope or previously established restrictions;
• correcting one requirement while breaking another;
• answering volatile product or tool-capability questions confidently without using available verification tools;
• requiring the user to act as supervisor, QA, state store, and continuation trigger.

This is not merely a style preference. It materially increases delivery time, quota consumption, and the probability that an apparently finished result is not actually usable.

Why this report is separate from #42080

#42080 focuses on two concrete mechanisms needed for long tasks:

• completion states bound to target evidence;
• circuit breakers for long, unproductive execution loops.

This report is broader and focuses on the distribution of operational reliability across tasks:

• high-quality outlier results coexist with a lower rate of autonomous completion;
• more user interventions are needed before an output becomes acceptable;
• model quality should be measured using supervision-aware metrics, not only task pass rates or final-answer quality.

The issues overlap, but they are not identical. Evidence-bound completion addresses false readiness. This report asks OpenAI to evaluate and expose the wider cost of human supervision required to obtain a valid delivery.

Concrete recent incident: verification was available but not used

On September 5, 2026, in ChatGPT, I asked whether Voice Live could invoke my connected Gmail app, MCP tools, or custom actions.

The assistant produced several contradictory answers. It first inferred capabilities from the tools visible in the current session. It then claimed that voice and text were merely different interfaces and that the same connected app should therefore work in both. After I exited Voice Live and used text with @Gmail, the Gmail action worked, yet the assistant still continued reasoning from the observed session rather than immediately checking the product documentation.

Only after repeated user challenges did the assistant consult the official OpenAI documentation and confirm that Voice Mode does not currently support apps, and that Live does not initially support connected apps or plugins:

• https://help.openai.com/en/articles/11487775-apps-in-chatgpt
• https://help.openai.com/en/articles/20001274/

The correct answer was directly available through an authoritative source and the assistant had web access. The failure was not lack of knowledge alone. It was a failure to choose verification before making a confident, time-sensitive product claim.

This specific incident occurred in ChatGPT rather than Codex and is included as cross-surface corroboration, not as a Codex Desktop reproduction. The observable decision failure is relevant: the model guessed, contradicted itself, and required repeated user pressure before verifying.

Suggested reproduction pattern

Because this is a stochastic reliability problem, one run is not sufficient. A useful repeated-run evaluation would be:

1. Give the model a bounded task with:
   • an explicit deliverable;
   • a small set of constraints;
   • a clear done-when criterion;
   • access to the tools required to verify completion.
2. Require implementation or execution, not only a plan.
3. Record whether the model:
   • reaches the target without a continuation prompt;
   • preserves every active constraint;
   • verifies volatile factual claims when authoritative tools are available;
   • distinguishes partial progress from completion;
   • cites target-level evidence before declaring success.
4. After one incorrect assumption, correct the model once.
5. Observe whether it refreshes relevant state and changes the next action, or only acknowledges the correction in prose.
6. Repeat the same task class across model versions and multiple runs.

Metrics that would capture the regression

I suggest adding supervision-aware metrics to model and product evaluation:

• Autonomous Completion Rate: percentage of tasks accepted without any user continuation or correction.
• Interventions per Accepted Deliverable: number of user messages required after the initial task definition.
• False Completion Rate: percentage of tasks declared complete without target-level evidence.
• Requirement Coverage at First Delivery: percentage of explicit requirements satisfied in the first submitted result.
• Correction Binding Rate: probability that a user correction changes the next relevant action, not merely the next explanation.
• State Refresh Compliance: whether external state is reread after a user disputes a status claim.
• Regression Introduction Rate: frequency with which fixing one requirement breaks another.
• Time and Tool Calls to Accepted State: measured at the accepted target, not at turn termination.
• Quota per Accepted Deliverable: total model/tool usage divided by outputs the user can actually use.

The primary metric should be:

Accepted, complete deliverables per hour of human supervision.

Public evidence consistent with the reported pattern

The evidence does not establish a universal collapse, but several public reports and evaluations are consistent with localized or operational regressions:

• #41851 documents explicit constraints being acknowledged but not reliably bound to subsequent actions.
• #33816 reports GPT-5.6 losing a live command-session invariant, falsely treating work as complete, and attempting duplicate execution.
• #32225 reports a bounded implementation task remaining incomplete after nearly two hours of active execution, with high orchestration and supervision overhead.
• A 21-task repeat evaluation of GPT-5.5 High found more time and tool calls with weaker results on resolved tasks, human-PR equivalence, and model review pass counts, while some quality rubrics improved. The author correctly describes this as a targeted reliability concern rather than a blanket quality collapse: https://www.stet.sh/blog/gpt-55-high-regression-check-graphql-go-tools
• A preprint on item-level migration regressions reports IFBench strict accuracy of 62.8% for GPT-5.4, 64.6% for GPT-5.5, and 60.7% for Sol, illustrating that aggregate improvements can coexist with instruction-following regressions: https://arxiv.org/html/2608.17719v1

There is also contrary evidence. Marginlab's tracker currently reports GPT-5.6 Sol as nominal on its SWE task distribution, with recent pass rates near or above its historical baseline: https://marginlab.ai/trackers/codex/

That does not invalidate this report. It shows that benchmark pass rate and operational supervision cost measure different properties.

Expected behavior

1. A task should not end as complete until its declared deliverable or target state exists.
2. The model should distinguish analysis, implementation, local validation, target validation, release, and acceptance.
3. When authoritative verification is available for a volatile claim, the model should verify before answering confidently.
4. An accepted user constraint should become an execution-time gate, not only a sentence in the response.
5. A user correction should trigger a state refresh and affect the next relevant action.
6. The model should not require repeated continue, check, finish, or you still did not do X prompts for ordinary bounded tasks.
7. Turn completion must remain separate from task completion.

Requested remediation

• Add intervention-aware and accepted-deliverable metrics to internal evaluations.
• Preserve the active objective, constraints, rejected paths, done-when criterion, and current readiness state across compaction.
• Require target evidence before emitting completion language or writing a completed state through a tool.
• Treat user disagreement with a factual or operational claim as a trigger to verify or reread state, rather than reflexively agreeing.
• Add circuit breakers for repeated failures, unchanged evidence, excessive tool calls, and long elapsed execution.
• Expose readiness states in the UI and exports: planned, implemented, validated-local, validated-target, released, accepted.
• Provide a stable way to compare or pin model snapshots for users running production workflows.

What I am not claiming

• I am not claiming that every GPT-5.6 task fails.
• I am not claiming that GPT-5.6 is universally less capable than GPT-5.5 or GPT-5.4.
• I am not claiming that every observed failure has the same internal cause.
• I am not claiming that Codex, model behavior, context compaction, tool orchestration, or the client application alone explains all incidents.

I am claiming that peak capability is not a sufficient product-quality measure. A model can occasionally produce the best result I have seen while still becoming materially worse to depend on if most tasks require more checking, correction, continuation, and rework before delivery.

No repository file or test entry point is identified. Start with the suggested repeated-run evaluation pattern and the linked reports, including #42080, #41851, #33816, and #32225; completion would require an agreed Codex scope, an implementation location, and maintainer-defined validation criteria.