anthropics / anthropics/claude-code

[BUG] /goal evaluator schema rejected by strict backends: required omits impossible

Open Beginner friendly
#95,021 0 comments 0 reactions 0 assignees View on GitHub
area:hooks area:providers bug has repro
Dominant language
Python
Stars
145k
Forks
23.1k
PR merge metrics
PR metrics pending

Description

## What's Wrong?

The `/goal` Stop-hook evaluator sends an `output_config.format` JSON schema whose `required` array omits one of its own properties (`impossible`). Any backend that enforces the strict schema subset (`required` must list every key in `properties`) rejects every evaluation with HTTP 400. Making this schema strict-subset compliant is free hardening: it changes nothing against Anthropic's own API and unbreaks every strict backend at once.

Reproduced against Meta Model API `/v1/messages` (Anthropic-compatible endpoint, model `muse-spark-1.3`): each turn that reaches Stop while a goal is active fails and the goal never evaluates.

The schema, from the 2.1.274 binary:

```json
{"type":"object","properties":{"ok":{"type":"boolean"},"reason":{"type":"string"},"impossible":{"type":"boolean"}},"required":["ok","reason"],"additionalProperties":false}
```

## What Should Happen?

The evaluator schema should be strict-compatible: `required` lists every key in `properties` (`["ok","reason","impossible"]`). In the 2.1.274 binary the verdict gate reads as a strict `=== true` check on `impossible`, so requiring the key should change no verdict logic.

## Error Messages/Logs

```text
Stop hook error: Hook evaluator API error: API Error: 400 'required' is required to be supplied and to be an array including every key in properties. Missing 'impossible'.
```

## Steps to Reproduce

Session-level:

1. Route Claude Code to Meta Model API (`ANTHROPIC_BASE_URL=https://api.meta.ai`, model `muse-spark-1.3`). Any strict-validating Anthropic-compatible backend applies.
2. Run `/goal` with any condition.
3. End a turn so it reaches Stop.
4. Observe the Stop hook error above. The goal never evaluates.

Backend-level (proves the schema, not the session, is at fault; key redacted):

```bash
# 400, exact message from the report above
curl https://api.meta.ai/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $META_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"muse-spark-1.3","max_tokens":50,"messages":[{"role":"user","content":"Reply with exactly: probe-ok"}],"output_config":{"effort":"max","format":{"type":"json_schema","schema":{"type":"object","properties":{"ok":{"type":"boolean"},"reason":{"type":"string"},"impossible":{"type":"boolean"}},"required":["ok","reason"],"additionalProperties":false}}}}'
```

```bash
# 200: identical request with "required":["ok","reason","impossible"]
curl https://api.meta.ai/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $META_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{"model":"muse-spark-1.3","max_tokens":50,"messages":[{"role":"user","content":"Reply with exactly: probe-ok"}],"output_config":{"effort":"max","format":{"type":"json_schema","schema":{"type":"object","properties":{"ok":{"type":"boolean"},"reason":{"type":"string"},"impossible":{"type":"boolean"}},"required":["ok","reason","impossible"],"additionalProperties":false}}}}'
```

A request with no `output_config` also returns 200, so auth, shape, and model are fine.

## Environment

- Claude Model: Other (`muse-spark-1.3` via third-party backend)
- Regression: I don't know
- Claude Code Version: 2.1.274 (updated from 2.1.273 today and re-verified the schema in the new binary before filing)
- Platform: Other (Meta Model API, Anthropic-compatible)
- Operating System: macOS
- Terminal: Terminal.app (macOS)

## Additional Information

- Searched open issues for `goal evaluator impossible required`: no duplicate found. Related but different cause: #80149 (400 on deferred MCP tool refs). Precedent for the fix direction: #37746 (closed completed, maintainer: fixed in next release; #54224 closed as its duplicate). That case was the same class with a different trigger: Vertex AI rejected the whole `output_config` param, while here a strict backend rejects a strict-subset violation. Same fix direction: client-side.
- Suggested fix: add `"impossible"` to `required` in the hook evaluator output schema.
- Preflight: searched existing issues, single bug, latest version.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the /goal Stop-hook evaluator and locate the output_config.format JSON schema described in the issue. Verify the strict-backend reproduction, then confirm that the evaluator accepts the response and the Stop evaluation completes successfully against a backend enforcing the schema subset.

Written by the indexing model from the issue text.

Assessment

Domain
api, cli
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.