anthropics / anthropics/claude-code
[FEATURE] Model self-assessment: expose a calibrated "this exceeds my tier" signal
- Lingua principale
- Python
- Stelle
- 145k
- Fork
- 23.1k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Preflight Checklist
- [x] I have searched existing requests and this feature hasn't been requested yet
- [x] This is a single feature request (not multiple features)
### Problem Statement
I run a four-tier delegation ladder across the model family — Haiku for mechanical legwork, Sonnet as the default drafting tier, Opus for hard reasoning and adversarial verification, Fable reserved for long-horizon work. Deciding *which tier a task belongs to* is a judgment I make by hand, before the task runs, on incomplete information. I get it wrong in both directions, and only one of those directions is visible to me.
Overspending is visible: I notice when Opus does something Haiku could have. Underspending is not. A model operating past its competence doesn't fail loudly — it returns a fluent, confident, plausible answer that happens to be wrong. There is no signal in the output distinguishing "this is right" from "this is past my depth." The burden of detecting the difference falls entirely on the user, who often lacks the expertise to catch it — that is usually *why* the task was delegated.
The result is that model selection is a guess the user makes before they have the information needed to make it, and a wrong guess is silent.
### Proposed Solution
Let the model report its own competence assessment for the task in front of it.
1. **A calibrated signal, exposed programmatically.** An API response field carrying a confidence-in-competence measure and, when it is low, a recommended tier. This is the part that matters most — it lets orchestrators route automatically instead of requiring a human to read a hint.
2. **Bidirectional.** "This exceeds my tier, consider Opus" *and* "this is well within a smaller model's range." Escalation alone becomes a cost ratchet.
3. **A brief inline note in interactive use**, not a blocking prompt: *"This involves multi-file refactoring with subtle state dependencies — Sonnet would handle this more reliably."*
4. **Silenceable.** A setting for users who don't want it.
The calibration bar is the whole feature. Naive implementations fail in both directions: fire constantly and users learn to ignore it (worse than nothing, since it launders overconfidence as diligence), or never fire and it is dead weight. A signal that is wrong is worse than absent, because it will be trusted.
### Alternative Solutions
What I do today, and why each falls short:
- **Convention encoded in CLAUDE.md** — a written delegation ladder with "escalate when a tier flags the work was too big for it." This works, but it depends on the cheap tier *noticing and admitting* it was out of its depth, which is exactly the capability being requested. The convention names the behavior; nothing produces it.
- **Verification passes** — have a higher tier adversarially check a lower tier's output. Reliable, but it pays the expensive model on every task to find the few that needed it, which inverts the savings.
- **Routing on task features** — keyword and complexity heuristics chosen in advance. Brittle; task difficulty is not legible from the prompt's surface.
**Related but distinct: #77986** ("Model-fluid delegation profile"), which asks for auto-tiered subagent routing with escalation on verification failure. That is orchestration-level, and its escalation trigger is *external* — something else must catch the defect first. This request is for the *internal* signal: the model reporting its own limits without a judge in the loop. The two compose well — a reliable self-assessment signal is what would let #77986's routing work without paying for a review pass on every task.
### Priority
Medium - Would be very helpful
### Feature Category
API and model interactions
### Use Case Example
1. An agentic pipeline dispatches a batch of ~200 code-comprehension tasks to Haiku for cost reasons.
2. Roughly a dozen involve cross-file state dependencies that Haiku handles unreliably.
3. Today: all 200 return confident answers. The dozen bad ones are indistinguishable from the rest, and surface later as defects, if ever.
4. With this feature: those dozen return a low-competence signal with a recommended tier. The orchestrator re-dispatches only those to Sonnet or Opus.
5. Cost stays near the Haiku baseline; the silent-failure class is largely eliminated. Neither a blanket upgrade nor a full verification pass achieves both.
### Additional Context
**There is already precedent in the product.** Anthropic support pointed me to [Why Claude switched models in your conversation with Opus 5](https://support.claude.com/en/articles/16049681-why-claude-switched-models-in-your-conversation-with-opus-5) — Opus 5 already performs automatic model switching for certain cybersecurity requests that fall outside its scope. The routing machinery exists; it is currently scoped to policy. This request is to aim the same mechanism at *capability*.
**On difficulty.** This is metacognition, and it is genuinely hard — models are known to be poorly calibrated about their own competence, and self-reported confidence correlates weakly with correctness. I would rather have a narrow signal that is well-calibrated on a few detectable task classes than a broad one that is noisy. A conservative version that only fires on high-confidence detections would still be valuable.
**On why it is worth doing anyway.** Overconfidence on hard tasks is a direct hallucination vector, and a model that accurately flags the edge of its competence is more credible on everything it does *not* flag. That is a trust property, and it is difficult for a competitor to copy quickly.
I sent a version of this to feedback@anthropic.com as well; filing here because the agentic/API framing is specific to this repo's users.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
No implementation file or test is named. Start by reading the existing model-switching behavior referenced in the issue and related issue #77986, then define the API contract, calibration scope, inline behavior, and silence setting; done means these requirements have an agreed, testable design.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Ambito
- ai, api
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Attiva
- Chiarezza
- Da chiarire
- Idoneità per principianti
- 24/100