vllm-project / vllm-project/vllm
[Bug]: guidance backend rejects tool-calling structural tags (KeyError: 'triggers') — strict/required/named tool_choice broken
- Dominant language
- Python
- Stars
- 91.8k
- Forks
- 22.2k
- PR merge metrics
- PR metrics pending
Description
### Your current environment
Verified on the official `vllm/vllm-openai:v0.28.0` Docker image (llguidance 1.7.6, xgrammar 0.2.3) and on the current nightly image (`vllm 0.28.1rc1.dev337+g27a94d1ce`, built 2026-09-03) — same failure, same line. The failure happens at request validation time, so it is model-, GPU- and traffic-independent.
### 🐛 Describe the bug
With `--structured-outputs-config '{"backend": "guidance"}'`, any tool-calling request that produces a structural tag is rejected before the first token with:
```
vllm.exceptions.VLLMValidationError: Invalid grammar specification: 'triggers'
```
This affects every request where the tool parser emits a structural tag, i.e. with a hermes-style parser:
- `tool_choice="auto"` with at least one `strict: true` tool,
- `tool_choice="required"`,
- named tool choice.
OpenAI-SDK clients commonly set `strict: true` on tools (pydantic-ai does it by default), so in practice tool calling is broken for those clients whenever the guidance backend is selected. Non-streaming requests get an HTTP 400; streaming requests get an in-stream error payload after the 200.
**Cause**: tool parsers build structural tags in the new xgrammar-style format, `{"type": "structural_tag", "format": {...}}` (via `vllm/tool_parsers/structural_tag_registry.py`), but `serialize_guidance_grammar` in `vllm/v1/structured_output/backend_guidance.py` only understands the legacy `{"triggers": [...], "structures": [...]}` shape and immediately does `s_tag["triggers"]` → `KeyError`, wrapped by `validate_guidance_grammar`. `backend_xgrammar.py` handles both formats (it branches on `"structures" in s_tag`); the guidance backend was never updated when the new format was introduced.
### Minimal reproduction (no GPU, no weights)
```python
import json
from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionToolsParam
from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
from vllm.v1.structured_output.backend_guidance import serialize_guidance_grammar
from vllm.v1.structured_output.backend_types import StructuredOutputOptions
tools = [ChatCompletionToolsParam(**{
"type": "function",
"function": {
"name": "get_weather",
"strict": True,
"parameters": {"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]},
},
})]
st = get_model_structural_tag("hermes", tools, "auto", False)
serialize_guidance_grammar(StructuredOutputOptions.STRUCTURAL_TAG, json.dumps(st.model_dump()))
```
```
Traceback (most recent call last):
File "/tmp/r.py", line 21, in
serialize_guidance_grammar(StructuredOutputOptions.STRUCTURAL_TAG, spec)
File ".../vllm/v1/structured_output/backend_guidance.py", line 267, in serialize_guidance_grammar
triggers: list[str] = s_tag["triggers"]
~~~~~^^^^^^^^^^^^
KeyError: 'triggers'
```
### Server-level reproduction
```bash
vllm serve Qwen/Qwen3-0.6B \
--enable-auto-tool-choice --tool-call-parser hermes \
--structured-outputs-config '{"backend": "guidance"}'
```
```bash
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "What is the weather in Paris?"}],
"tools": [{"type": "function", "function": {"name": "get_weather", "strict": true,
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
"tool_choice": "auto"
}'
```
→ `{"error":{"message":"Invalid grammar specification: 'triggers'", ...}}` (the same request without `"strict": true` works). `tool_choice="required"` and named tool choice fail identically, even without `strict`.
### Related finding
While fixing this we noticed the legacy path has a second problem: it passes the output of `LLMatcher.grammar_from_json_schema` (a serialized `{"grammars": [...]}` envelope) as `llguidance.StructTag.grammar`, which llguidance silently interprets as an empty JSON schema — so tool-call arguments were effectively unconstrained even when the legacy format worked.
I have a fix that translates the new-format structural tags (`triggered_tags`, `tags_with_separator`, `any_text`, as emitted by the tool-parser registry) into llguidance grammars and rejects unsupported shapes with a clear error; it has been running in production for us. PR coming shortly, with the legacy-path schema fix as a separate commit (easy to split out if preferred).
Contributor guide
Research direction
Start with vllm/v1/structured_output/backend_guidance.py, especially serialize_guidance_grammar, and compare its structural-tag handling with backend_xgrammar.py and structural_tag_registry.py. Run the minimal reproduction, then verify that strict auto, required, and named tool choices no longer fail with KeyError and that tool-call arguments remain constrained by their schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100