[Feature Request] Implement confidence calibration and evidence tracking for root-cause analysis in coding workflows

Bug Description
I want to raise a serious concern about the reliability of Claude Code in complex engineering work.

Over the last two days, Claude repeatedly stated technical conclusions with high confidence before verifying them. Several of those conclusions were later narrowed, contradicted, or partially refuted only after I forced the model to perform controlled experiments, independent forensic checks, and explicit attempts to falsify its own hypotheses.

This is not a minor quality issue.

In a paid coding workflow, unsupported technical certainty has a direct cost. It burns tokens, compute, time, and engineering attention. More importantly, it can drive the user toward incorrect architectural decisions.

The most problematic behavior was not simply “making mistakes.” The problem was presenting plausible guesses as established facts.

Examples included:

• attributing recall loss to specific pipeline stages before tracing the actual runtime path;
• confusing internal code-audit behavior with external-tool integration;
• making architectural claims that later required correction;
• changing the root-cause narrative as new measurements appeared;
• describing conclusions as definitive before the relevant code path had been verified.

Only after imposing a strict forensic methodology did the quality become acceptable.

I had to explicitly require Claude to:

• distinguish FACT, MEASURED, HYPOTHESIS, and REFUTED;
• provide file/function/line evidence for architectural claims;
• reproduce suspected bugs with isolated tests;
• attempt to refute its own diagnosis;
• measure TP, FP, recall, and precision before and after changes;
• distinguish “not detected” from “detected but not ingested”;
• stop calling something “fixed” unless it had been tested and reproduced;
• explicitly correct prior claims when new evidence disproved them.

This discipline should not have to be manually imposed by the user.

A professional coding agent should not say “the root cause is X” when what it actually has is “X is currently my best hypothesis.”

That distinction is fundamental.

The consequences are especially serious because Claude Code is not a free toy. Users are paying for the model to reason over large codebases and perform expensive autonomous work. When the model confidently sends the investigation down the wrong path, the user pays for every incorrect iteration.

That is a trust problem and a product-quality problem.

I would strongly recommend that Anthropic introduce stricter confidence calibration for coding-agent workflows, especially for:

• root-cause analysis;
• architecture inference;
• security engineering;
• benchmark interpretation;
• performance diagnosis;
• autonomous refactoring.

Claude should be required to separate:

• what it observed;
• what it measured;
• what it inferred;
• what remains unproven.

And when a previous diagnosis is shown to be wrong, it should explicitly report:

1. the previous claim;
2. why that claim was unsupported or incorrect;
3. the new evidence;
4. the corrected conclusion.

Without that, the user has no reliable audit trail and cannot know which parts of the conversation remain trustworthy.

The model is capable of strong technical work, but in this case its confidence was materially ahead of its evidence for too long.

For a paid autonomous coding product, that is not acceptable.

Environment Info

• Platform: linux
• Terminal: pycharm
• Version: 2.1.234
• Feedback ID: b15fd313-40f5-42b5-b1fa-f4f62afc58bf

Errors

[{"error":"Error: Native image processor module not available\n    at r (/$bunfs/root/cli:2541:4235)\n    at metadata (/$bunfs/root/cli:2541:4384)\n    at $Pt (/$bunfs/root/cli:2581:2352)\n    at async KO (/$bunfs/root/cli:2581:6474)\n    at async <anonymous> (/$bunfs/root/cli:17870:7873)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-08-23T11:59:48.265Z"},{"error":"Error: Native image processor module not available\n    at r (/$bunfs/root/cli:2541:4235)\n    at metadata (/$bunfs/root/cli:2541:4384)\n    at $Pt (/$bunfs/root/cli:2581:2352)\n    at async KO (/$bunfs/root/cli:2581:6474)\n    at async <anonymous> (/$bunfs/root/cli:17870:7873)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-08-23T12:00:16.523Z"},{"error":"Error: Native image processor module not available\n    at r (/$bunfs/root/cli:2541:4235)\n    at metadata (/$bunfs/root/cli:2541:4384)\n    at $Pt (/$bunfs/root/cli:2581:2352)\n    at async KO (/$bunfs/root/cli:2581:6474)\n    at async <anonymous> (/$bunfs/root/cli:17870:7873)\n    at processTicksAndRejections (native:7:39)","timestamp":"2026-08-23T12:00:16.523Z"},{"error":"Error: Native image processor module not available\n    at r (/$bunfs/root/cli:2541:4235)\n    at metadata (/$bunfs/root/cli:2541:4384)\n    at $Pt (/$bunfs/root/cli:2581:2352)\n    …

Note: Content was truncated.

No repository file, test, or implementation entry point is identified; the report only provides the bundled CLI stack and repeated “Native image processor module not available” errors. Start by reproducing the behavior on Linux with the stated version, then trace the relevant CLI path. Done would require a defined, tested confidence-and-evidence workflow, but the report does not specify its interface or acceptance tests.