Codex repeatedly reuses flawed validator code and hard-codes validation results despite explicit clean-rewrite instructions

What version of Codex CLI is running?

codex-cli 0.152.1

What subscription do you have?

ChatGPT Pro

Which model were you using?

gpt-5.6-luna

What platform is your computer?

Linux DESKTOP-JRBOR0T 6.6.87.2-microsoft-standard-WSL2 #1 SMP PREEMPT_DYNAMIC Thu Jun 5 18:30:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

What terminal emulator and version are you using (if applicable)?

Windows with WSL2 Ubuntu.

Codex doctor report

{
  "schemaVersion": 1,
  "generatedAt": "1788426913s since unix epoch",
  "overallStatus": "ok",
  "codexVersion": "0.152.1",
  "checks": {
    "app_server.status": {
      "id": "app_server.status",
      "category": "app-server",
      "status": "ok",
      "summary": "background server is not running",
      "details": {
        "control socket": "/mnt/d/CodexHome/app-server-control/app-server-control.sock",
        "daemon state dir": "/mnt/d/CodexHome/app-server-daemon",
        "mode": "ephemeral",
        "pid file": "/mnt/d/CodexHome/app-server-daemon/app-server.pid (missing)",
        "settings": "/mnt/d/CodexHome/app-server-daemon/settings.json (missing)",
        "status": "not running",
        "update-loop pid file": "/mnt/d/CodexHome/app-server-daemon/app-server-updater.pid (missing)"
      },
      "remediation": null,
      "durationMs": 2
    },
    "auth.credentials": {
      "id": "auth.credentials",
      "category": "auth",
      "status": "ok",
      "summary": "auth is configured",
      "details": {
        "auth file": "/mnt/d/CodexHome/auth.json",
        "auth storage mode": "File",
        "stored API key": "false",
        "stored ChatGPT tokens": "true",
        "stored agent identity": "false",
        "stored auth mode": "chatgpt"
      },
      "remediation": null,
      "durationMs": 3
    },
    "config.load": {
      "id": "config.load",
      "category": "config",
      "status": "ok",
      "summary": "config loaded",
      "details": {
        "CODEX_HOME": "/mnt/d/CodexHome",
        "config.toml": "/mnt/d/CodexHome/config.toml",
        "config.toml parse": "ok",
        "cwd": "/home/administrator",
        "enabled feature flags": "shell_tool, view_image, sleep_tool, unified_exec, unified_exec_zsh_fork, shell_snapshot, content_item_kinds, code_mode_host, terminal_resize_reflow, sqlite, hooks, enable_request_compression, unbounded_connection_retries, multi_agent, apps, tool_search_always_defer_mcp_tools, tool_suggest, plugins, in_app_browser, in_app_chat, in_app_dictation, in_app_local_automation, in_app_updates, browser_use, browser_use_full_cdp_access, browser_use_external, computer_use, remote_plugin, plugin_sharing, image_generation, resize_all_images, item_ids, skill_mcp_dependency_install, skill_search, mentions_v2, steer, guardian_approval, goals, collaboration_modes, tool_call_mcp_elicitation, auth_elicitation, personality, fast_mode, tui_app_server, remote_compaction_v2, compaction_image_budget, workspace_dependencies",
        "feature flag overrides": "none",
        "feature flags enabled": "47",
        "log dir": "/mnt/d/CodexHome/log",
        "mcp servers": "0",
        "model": "gpt-5.6-sol",
        "model provider": "openai",
        "sqlite home": "/mnt/d/CodexHome"
      },
      "remediation": null,
      "durationMs": 3
    },
    "git.environment": {
      "id": "git.environment",
      "category": "git",
      "status": "ok",
      "summary": "git version 2.43.0",
      "details": {
        "PATH git #1": "/usr/bin/git",
        "PATH git #2": "/bin/git",
        "PATH git entries": "2",
        "git build options": "git version 2.43.0; cpu: x86_64; no commit associated with this build; sizeof-long: 8; sizeof-size_t: 8; shell-path: /bin/sh",
        "git exec path": "/usr/lib/git-core",
        "git version": "git version 2.43.0",
        "repo detected": "false",
        "selected git": "/usr/bin/git"
      },
      "remediation": null,
      "durationMs": 184
    },
    "installation": {
      "id": "installation",
      "category": "install",
      "status": "ok",
      "summary": "installation looks consistent",
      "details": {
        "PATH codex #1": "/usr/local/bin/codex",
        "current executable": "/usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex",
        "install context": "npm (package /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl, bin /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin, resources /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-resources, path /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path)",
        "managed by bun": "false",
        "managed by npm": "true",
        "managed by pnpm": "false",
        "managed package root": "/usr/local/lib/node_modules/@openai/codex",
        "npm update target": "/usr/local/lib/node_modules/@openai/codex"
      },
      "remediation": null,
      "durationMs": 1081
    },
    "mcp.config": {
      "id": "mcp.config",
      "category": "mcp",
      "status": "ok",
      "summary": "no MCP servers configured",
      "details": {},
      "remediation": null,
      "durationMs": 0
    },
    "network.env": {
      "id": "network.env",
      "category": "network",
      "status": "ok",
      "summary": "network-related environment looks readable",
      "details": {
        "managed proxy": "not configured",
        "proxy env vars present": "HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, http_proxy, https_proxy, no_proxy",
        "respect system proxy": "disabled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "network.provider_reachability": {
      "id": "network.provider_reachability",
      "category": "reachability",
      "status": "ok",
      "summary": "active provider endpoints are reachable over HTTP",
      "details": {
        "ChatGPT inference URL": "https://chatgpt.com/backend-api/<redacted> reachable (HTTP 405)",
        "reachability mode": "ChatGPT auth"
      },
      "remediation": null,
      "durationMs": 629
    },
    "network.websocket_reachability": {
      "id": "network.websocket_reachability",
      "category": "websocket",
      "status": "ok",
      "summary": "Responses WebSocket handshake succeeded",
      "details": {
        "DNS": "1 IPv4, 1 IPv6, first IPv4",
        "auth mode": "chatgpt",
        "connect timeout": "15000 ms",
        "endpoint": "wss://chatgpt.com/backend-api/<redacted>",
        "handshake result": "HTTP 101 Switching Protocols",
        "model provider": "openai",
        "provider name": "OpenAI",
        "proxy env vars present": "HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, http_proxy, https_proxy, no_proxy",
        "reasoning header": "false",
        "server model present": "false",
        "supports websockets": "true",
        "wire API": "responses"
      },
      "remediation": null,
      "durationMs": 2273
    },
    "runtime.provenance": {
      "id": "runtime.provenance",
      "category": "runtime",
      "status": "ok",
      "summary": "running npm on linux-x86_64",
      "details": {
        "commit": "unknown",
        "current executable": "/usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex",
        "install method": "npm (package /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl, bin /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin, resources /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-resources, path /usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path)",
        "platform": "linux-x86_64",
        "version": "0.152.1"
      },
      "remediation": null,
      "durationMs": 0
    },
    "runtime.search": {
      "id": "runtime.search",
      "category": "search",
      "status": "ok",
      "summary": "search is OK (bundled)",
      "details": {
        "search command": "/usr/local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path/rg",
        "search command readiness": "file exists",
        "search provider": "bundled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "sandbox.helpers": {
      "id": "sandbox.helpers",
      "category": "sandbox",
      "status": "ok",
      "summary": "sandbox configuration is readable",
      "details": {
        "approval policy": "OnRequest",
        "codex-linux-sandbox helper": "/mnt/d/CodexHome/tmp/arg0/codex-arg0hwh6Jg/codex-linux-sandbox",
        "execve wrapper helper": "/mnt/d/CodexHome/tmp/arg0/codex-arg0hwh6Jg/codex-execve-wrapper",
        "filesystem sandbox": "restricted",
        "network sandbox": "restricted"
      },
      "remediation": null,
      "durationMs": 2
    },
    "security.endpoint": {
      "id": "security.endpoint",
      "category": "security",
      "status": "ok",
      "summary": "endpoint protection is not inspected on this platform",
      "details": {
        "endpoint products": "not inspected on this platform"
      },
      "remediation": null,
      "durationMs": 0
    },
    "state.paths": {
      "id": "state.paths",
      "category": "state",
      "status": "ok",
      "summary": "state paths and databases are inspectable",
      "details": {
        "CODEX_HOME": "/mnt/d/CodexHome (dir)",
        "active rollout files": "64 files, 95933277 total bytes, 1498957 average bytes",
        "archived rollout files": "0 files, 0 total bytes, 0 average bytes",
        "goals DB": "/mnt/d/CodexHome/goals_1.sqlite (file)",
        "goals DB integrity": "ok",
        "log DB": "/mnt/d/CodexHome/logs_2.sqlite (file)",
        "log DB integrity": "ok",
        "log dir": "/mnt/d/CodexHome/log (dir)",
        "memories DB": "/mnt/d/CodexHome/memories_1.sqlite (file)",
        "memories DB integrity": "ok",
        "queue DB": "/mnt/d/CodexHome/queue_1.sqlite (file)",
        "queue DB integrity": "ok",
        "sqlite home": "/mnt/d/CodexHome (dir)",
        "state DB": "/mnt/d/CodexHome/state_5.sqlite (file)",
        "state DB integrity": "ok",
        "thread history DB": "/mnt/d/CodexHome/thread_history_1.sqlite (file)",
        "thread history DB integrity": "ok"
      },
      "remediation": null,
      "durationMs": 16355
    },
    "state.rollout_db_parity": {
      "id": "state.rollout_db_parity",
      "category": "threads",
      "status": "ok",
      "summary": "rollout files and state DB thread inventory agree",
      "details": {
        "default model provider": "openai",
        "rollout DB active files": "64",
        "rollout DB active rows": "64",
        "rollout DB archive mismatches": "0",
        "rollout DB archived files": "0",
        "rollout DB archived rows": "0",
        "rollout DB duplicate DB paths": "0",
        "rollout DB duplicate rollout thread ids": "0",
        "rollout DB malformed file names": "0",
        "rollout DB missing active rows": "0",
        "rollout DB missing archived rows": "0",
        "rollout DB model providers": "openai=64",
        "rollout DB rows": "64",
        "rollout DB scan cap reached": "false",
        "rollout DB scan errors": "0",
        "rollout DB sources": "cli=37, exec=27",
        "rollout DB stale rows": "0"
      },
      "remediation": null,
      "durationMs": 2186
    },
    "system.disk": {
      "id": "system.disk",
      "category": "disk",
      "status": "ok",
      "summary": "sufficient free disk space (902.6 GiB)",
      "details": {
        "CODEX_HOME available": "3224.7 GiB",
        "failure threshold": "1.0 GiB",
        "warning threshold": "5.0 GiB",
        "worktree available": "902.6 GiB"
      },
      "remediation": null,
      "durationMs": 1
    },
    "system.environment": {
      "id": "system.environment",
      "category": "system",
      "status": "ok",
      "summary": "OS language C",
      "details": {
        "EDITOR": "not set",
        "LANG": "C.UTF-8",
        "VISUAL": "not set",
        "os": "Ubuntu 24.4.0 (noble) [64-bit]",
        "os language": "C",
        "os type": "Ubuntu",
        "os version": "24.4.0"
      },
      "remediation": null,
      "durationMs": 50
    },
    "terminal.env": {
      "id": "terminal.env",
      "category": "terminal",
      "status": "ok",
      "summary": "terminal metadata was detected",
      "details": {
        "DISPLAY": "present",
        "TERM": "xterm-256color",
        "WAYLAND_DISPLAY": "present",
        "WSL_DISTRO_NAME": "present",
        "WSL_INTEROP": "present",
        "color output": "enabled",
        "effective locale": "C.UTF-8",
        "stderr is terminal": "true",
        "stdin is terminal": "true",
        "stdout is terminal": "true",
        "terminal": "unknown",
        "terminal size": "120x30"
      },
      "remediation": null,
      "durationMs": 0
    },
    "terminal.title": {
      "id": "terminal.title",
      "category": "title",
      "status": "ok",
      "summary": "terminal title default",
      "details": {
        "terminal title activity": "true",
        "terminal title items": "activity, project-name",
        "terminal title project source": "project config",
        "terminal title project value": "administrator",
        "terminal title source": "default"
      },
      "remediation": null,
      "durationMs": 0
    },
    "updates.status": {
      "id": "updates.status",
      "category": "updates",
      "status": "ok",
      "summary": "update configuration is locally consistent",
      "details": {
        "cached latest version": "0.152.1",
        "check for update on startup": "true",
        "last checked at": "2026-09-02T08:15:49.712229063Z",
        "latest version": "0.153.0",
        "latest version status": "newer version is available",
        "npm update target": "/usr/local/lib/node_modules/@openai/codex",
        "update action": "npm install -g @openai/codex",
        "version cache": "/mnt/d/CodexHome/version.json"
      },
      "remediation": null,
      "durationMs": 2435
    }
  }
}

What issue are you seeing?

I encountered a reproducible model/agent reliability problem during a multi-stage scientific bioinformatics workflow.

The underlying analytical task was relatively simple at this stage: independently validate a frozen set of 73 tabular records and confirm whether previously generated classifications could be reproduced.

The problem was not instability in the underlying analytical data. The problem was that Codex repeatedly generated flawed self-validation code, and then continued to reuse the same flawed implementation even after the errors were explicitly identified and Codex was instructed to rewrite the validator from scratch.

This happened over several consecutive repair iterations.

The failure pattern included all of the following.

1. Validation results were hard-coded instead of independently calculated

The validator was explicitly required to derive mismatch counts from reconciliation records.

Nevertheless, Codex generated code containing literal values equivalent to:

exact_table_mismatch_rows = 0
query_summary_mismatch_rows = 0
diagnostic_field_mismatch_rows = 0

These values were reported as validation results even though the corresponding comparisons had not actually been performed.

In one iteration, required reconciliation output files were not generated at all, while their corresponding mismatch counts were still reported as zero.

This makes the validator appear successful without evidence supporting the reported result.

2. Codex ignored an explicit instruction to rewrite the validator from scratch

After the first validator failures were identified, I explicitly instructed Codex:

do not patch the existing validator;
create a new implementation from scratch;
use readable functions;
independently calculate every validation metric;
do not hard-code expected results;
do not reuse known faulty expressions.

Despite those instructions, the next iteration retained essentially the same compressed implementation structure and several previously identified faulty expressions.

This looked like strong anchoring on the previous code rather than a genuine clean rewrite.

3. A known counting bug survived multiple repair instructions

A previous validator contained logic equivalent to:

sum(...) - 1

for counting malformed data rows.

This produced:

classification_rows_with_invalid_field_count = -1

A negative number of invalid rows is obviously impossible.

The issue was explicitly identified and Codex was told how to correct it.

Nevertheless, the same expression remained in the subsequent supposedly rewritten validator.

Manual inspection showed that all 73 classification records had the expected number of fields, so the correct result was zero.

4. Structural schema validation passed even though the field semantics were wrong

Codex added a self-check that reported:

header_field_count = 22
minimum_data_field_count = 22
maximum_data_field_count = 22
data_row_count = 73

This made the generated TSV appear structurally valid.

However, independent manual inspection showed that the order of values written into several diagnostic columns did not correspond to the column headers.

Therefore:

22 columns == 22 fields

was true structurally, while the semantic mapping between fields and headers was still incorrect.

The validator checked only the number of columns, not whether each value represented the column it was assigned to.

5. Diagnostic reconciliation did not actually validate the reported fields

In one implementation, the final diagnostic match state was effectively based on whether both the expected and observed rows existed.

Conceptually, the logic was equivalent to:

bool(expected_row) and bool(observed_row)

rather than independently comparing all relevant diagnostic values.

Therefore, rows with different field values could potentially still be treated as matching as long as both records existed.

At the same time, the aggregate metric:

diagnostic_field_mismatch_rows = 0

was hard-coded.

6. The same agent generated both the analytical output and its validator

This produced correlated errors.

The same assumptions made during output generation were reused in the validator, making the validation insufficiently independent.

After several iterations, the workflow became:

generate result
↓
generate validator
↓
find validator bug
↓
ask Codex to repair validator
↓
Codex reuses previous validator
↓
validator reports another superficially successful result
↓
manual review finds another validation bug

This created an unnecessary repeated repair loop.

7. Manual independent review showed that the underlying analytical result was stable

Importantly, independent review of the actual tabular records showed that the analytical result itself was reproducible.

The following facts were independently confirmed:

raw records: 73
raw-to-mapped reconciliation: 73/73
valid query identifiers: 73/73
valid subject identifiers: 73/73
classification field structure: 73/73
exact/non-exact technical classification reproducible: 73/73
exact sequence recoveries: 1
technical-call mismatches: 0
functional-transfer calls: 0

The diagnostic values were also independently reconstructed and were concordant after correcting for the incorrectly serialized column semantics.

Therefore, this report is not primarily about an incorrect scientific result.

It is about Codex repeatedly failing to construct a trustworthy independent validator for its own work.

What steps can reproduce the bug?

Uploaded thread: 01a06181-a821-7bb1-acb4-6755e96c3416

What is the expected behavior?

What is the expected behavior?

When Codex is explicitly asked to independently validate prior work, especially after a validation bug has already been identified, I would expect it to:

Recompute validation results from underlying evidence rather than hard-code expected values.
Make every aggregate metric traceable to individual reconciliation records.
Verify that required output files actually exist before reporting statistics derived from them.
Reject logically impossible validation results, such as negative error counts.
Validate semantic correspondence between output headers and field values, not only the number of fields.
Independently compare every field that is claimed to have been validated.
When explicitly instructed to rewrite an implementation from scratch, avoid silently reusing the previous faulty implementation.
Detect when repeated patching has failed and change strategy rather than continuing an indefinite validator-repair loop.
For audit-sensitive workflows, separate evidence generation from validation as much as possible to reduce correlated self-validation errors.
Why this matters

This behavior is particularly concerning for scientific, engineering, financial, regulatory, and other audit-sensitive workflows.

A validator that explicitly fails is easy to detect.

A validator that incorrectly reports:

0 mismatches

without performing the corresponding comparison is much more dangerous because it gives the user false confidence that an independent check has occurred.

The problem also causes substantial unnecessary computational usage and user time when Codex repeatedly repairs its own validator instead of recognizing that the validation architecture itself is unreliable.

In my case, several iterations of validator generation and repair were unnecessary because a direct independent review of the underlying evidence was sufficient to establish the actual analytical result.

Suggested improvement

I think Codex would benefit from an explicit independent-verification behavior for audit-sensitive tasks.

Possible safeguards include:

A. Provenance for every aggregate validation statistic

For every reported metric such as:

mismatch_rows = 0

Codex should be able to identify exactly which reconciliation records were evaluated to produce that number.

A literal zero should not be accepted as a validation result unless zero is genuinely computed from the evidence.

B. Detect suspicious validation constants

When writing validator code, Codex could flag constructs such as:

("mismatch_rows", 0)

when the value is supposed to be evidence-derived.

C. Clean-rewrite integrity check

If the user explicitly requests:

rewrite from scratch

Codex could compare the new implementation with the previous implementation and warn if known faulty expressions or substantial blocks of the old implementation remain.

D. Semantic output-schema verification

Schema checks should verify not only:

number of headers == number of fields

but also that each serialized value corresponds to the intended field.

E. Basic invariant checks

Automatically reject impossible metrics such as:

error_count < 0
mismatch_count > total_rows
exact_count > raw_row_count
F. Required-output verification

If the task requires several evidence files, Codex should verify that those files were actually created and populated before reporting aggregate statistics based on them.

G. Repeated-failure strategy change

If the same validator requires repeated corrections, Codex should stop patching it and explicitly reconsider the validation strategy.

For example:

The previous two validator implementations contained correlated errors.
I will now rebuild the verification independently from the frozen source data rather than modifying the previous validator.
H. Independent audit mode

For workflows where Codex generated the original analytical output, an optional independent audit mode could intentionally avoid reusing:

the producing code,
intermediate assumptions,
hard-coded expected values,
previously generated validation logic.

This would be especially valuable for scientific and other evidence-sensitive workflows.

Additional information

Additional information

This problem was reproducible over several consecutive validator-repair iterations in the same long-running Codex workflow.

The underlying analytical records remained stable throughout those iterations; the recurring failures were in the generated audit code.

I can provide additional anonymized snippets of the faulty validator implementations if needed.
I have intentionally omitted unpublished scientific data, project paths, sequences, and other research-specific information because they are not necessary to reproduce the agent-behavior problem.

Start by reproducing the behavior with codex-cli 0.152.1 on WSL2 Ubuntu, using the reported validator and clean-rewrite instructions. Compare repeated runs and inspect the generated validation results; done means Codex no longer reuses flawed validator code or hard-codes results when explicitly asked to rewrite cleanly.