anthropics / anthropics/claude-code
[BUG] /goal evaluator schema rejected by strict backends: required omits impossible
- Dominant language
- Python
- Stars
- 145k
- Forks
- 23.1k
- PR merge metrics
- PR metrics pending
Description
## What's Wrong?
The `/goal` Stop-hook evaluator sends an `output_config.format` JSON schema whose `required` array omits one of its own properties (`impossible`). Any backend that enforces the strict schema subset (`required` must list every key in `properties`) rejects every evaluation with HTTP 400. Making this schema strict-subset compliant is free hardening: it changes nothing against Anthropic's own API and unbreaks every strict backend at once.
Reproduced against Meta Model API `/v1/messages` (Anthropic-compatible endpoint, model `muse-spark-1.3`): each turn that reaches Stop while a goal is active fails and the goal never evaluates.
The schema, from the 2.1.274 binary:
```json
{"type":"object","properties":{"ok":{"type":"boolean"},"reason":{"type":"string"},"impossible":{"type":"boolean"}},"required":["ok","reason"],"additionalProperties":false}
```
## What Should Happen?
The evaluator schema should be strict-compatible: `required` lists every key in `properties` (`["ok","reason","impossible"]`). In the 2.1.274 binary the verdict gate reads as a strict `=== true` check on `impossible`, so requiring the key should change no verdict logic.
## Error Messages/Logs
```text
Stop hook error: Hook evaluator API error: API Error: 400 'required' is required to be supplied and to be an array including every key in properties. Missing 'impossible'.
```
## Steps to Reproduce
Session-level:
1. Route Claude Code to Meta Model API (`ANTHROPIC_BASE_URL=https://api.meta.ai`, model `muse-spark-1.3`). Any strict-validating Anthropic-compatible backend applies.
2. Run `/goal` with any condition.
3. End a turn so it reaches Stop.
4. Observe the Stop hook error above. The goal never evaluates.
Backend-level (proves the schema, not the session, is at fault; key redacted):
```bash
# 400, exact message from the report above
curl https://api.meta.ai/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $META_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"muse-spark-1.3","max_tokens":50,"messages":[{"role":"user","content":"Reply with exactly: probe-ok"}],"output_config":{"effort":"max","format":{"type":"json_schema","schema":{"type":"object","properties":{"ok":{"type":"boolean"},"reason":{"type":"string"},"impossible":{"type":"boolean"}},"required":["ok","reason"],"additionalProperties":false}}}}'
```
```bash
# 200: identical request with "required":["ok","reason","impossible"]
curl https://api.meta.ai/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $META_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"muse-spark-1.3","max_tokens":50,"messages":[{"role":"user","content":"Reply with exactly: probe-ok"}],"output_config":{"effort":"max","format":{"type":"json_schema","schema":{"type":"object","properties":{"ok":{"type":"boolean"},"reason":{"type":"string"},"impossible":{"type":"boolean"}},"required":["ok","reason","impossible"],"additionalProperties":false}}}}'
```
A request with no `output_config` also returns 200, so auth, shape, and model are fine.
## Environment
- Claude Model: Other (`muse-spark-1.3` via third-party backend)
- Regression: I don't know
- Claude Code Version: 2.1.274 (updated from 2.1.273 today and re-verified the schema in the new binary before filing)
- Platform: Other (Meta Model API, Anthropic-compatible)
- Operating System: macOS
- Terminal: Terminal.app (macOS)
## Additional Information
- Searched open issues for `goal evaluator impossible required`: no duplicate found. Related but different cause: #80149 (400 on deferred MCP tool refs). Precedent for the fix direction: #37746 (closed completed, maintainer: fixed in next release; #54224 closed as its duplicate). That case was the same class with a different trigger: Vertex AI rejected the whole `output_config` param, while here a strict backend rejects a strict-subset violation. Same fix direction: client-side.
- Suggested fix: add `"impossible"` to `required` in the hook evaluator output schema.
- Preflight: searched existing issues, single bug, latest version.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the /goal Stop-hook evaluator and locate the output_config.format JSON schema described in the issue. Verify the strict-backend reproduction, then confirm that the evaluator accepts the response and the Stop evaluation completes successfully against a backend enforcing the schema subset.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, cli
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100