vllm-project / vllm-project/vllm

[Bug]: guidance backend rejects tool-calling structural tags (KeyError: 'triggers') — strict/required/named tool_choice broken

Open
#55,152 3 comments 0 reactions 0 assignees View on GitHub
structured-output tool-calling
Dominant language
Python
Stars
91.8k
Forks
22.2k
PR merge metrics
PR metrics pending

Description

### Your current environment

Verified on the official `vllm/vllm-openai:v0.28.0` Docker image (llguidance 1.7.6, xgrammar 0.2.3) and on the current nightly image (`vllm 0.28.1rc1.dev337+g27a94d1ce`, built 2026-09-03) — same failure, same line. The failure happens at request validation time, so it is model-, GPU- and traffic-independent.

### 🐛 Describe the bug

With `--structured-outputs-config '{"backend": "guidance"}'`, any tool-calling request that produces a structural tag is rejected before the first token with:

```
vllm.exceptions.VLLMValidationError: Invalid grammar specification: 'triggers'
```

This affects every request where the tool parser emits a structural tag, i.e. with a hermes-style parser:

- `tool_choice="auto"` with at least one `strict: true` tool,
- `tool_choice="required"`,
- named tool choice.

OpenAI-SDK clients commonly set `strict: true` on tools (pydantic-ai does it by default), so in practice tool calling is broken for those clients whenever the guidance backend is selected. Non-streaming requests get an HTTP 400; streaming requests get an in-stream error payload after the 200.

**Cause**: tool parsers build structural tags in the new xgrammar-style format, `{"type": "structural_tag", "format": {...}}` (via `vllm/tool_parsers/structural_tag_registry.py`), but `serialize_guidance_grammar` in `vllm/v1/structured_output/backend_guidance.py` only understands the legacy `{"triggers": [...], "structures": [...]}` shape and immediately does `s_tag["triggers"]` → `KeyError`, wrapped by `validate_guidance_grammar`. `backend_xgrammar.py` handles both formats (it branches on `"structures" in s_tag`); the guidance backend was never updated when the new format was introduced.

### Minimal reproduction (no GPU, no weights)

```python
import json
from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionToolsParam
from vllm.tool_parsers.structural_tag_registry import get_model_structural_tag
from vllm.v1.structured_output.backend_guidance import serialize_guidance_grammar
from vllm.v1.structured_output.backend_types import StructuredOutputOptions

tools = [ChatCompletionToolsParam(**{
"type": "function",
"function": {
"name": "get_weather",
"strict": True,
"parameters": {"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]},
},
})]

st = get_model_structural_tag("hermes", tools, "auto", False)
serialize_guidance_grammar(StructuredOutputOptions.STRUCTURAL_TAG, json.dumps(st.model_dump()))
```

```
Traceback (most recent call last):
File "/tmp/r.py", line 21, in
serialize_guidance_grammar(StructuredOutputOptions.STRUCTURAL_TAG, spec)
File ".../vllm/v1/structured_output/backend_guidance.py", line 267, in serialize_guidance_grammar
triggers: list[str] = s_tag["triggers"]
~~~~~^^^^^^^^^^^^
KeyError: 'triggers'
```

### Server-level reproduction

```bash
vllm serve Qwen/Qwen3-0.6B \
--enable-auto-tool-choice --tool-call-parser hermes \
--structured-outputs-config '{"backend": "guidance"}'
```

```bash
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [{"role": "user", "content": "What is the weather in Paris?"}],
"tools": [{"type": "function", "function": {"name": "get_weather", "strict": true,
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
"tool_choice": "auto"
}'
```

→ `{"error":{"message":"Invalid grammar specification: 'triggers'", ...}}` (the same request without `"strict": true` works). `tool_choice="required"` and named tool choice fail identically, even without `strict`.

### Related finding

While fixing this we noticed the legacy path has a second problem: it passes the output of `LLMatcher.grammar_from_json_schema` (a serialized `{"grammars": [...]}` envelope) as `llguidance.StructTag.grammar`, which llguidance silently interprets as an empty JSON schema — so tool-call arguments were effectively unconstrained even when the legacy format worked.

I have a fix that translates the new-format structural tags (`triggered_tags`, `tags_with_separator`, `any_text`, as emitted by the tool-parser registry) into llguidance grammars and rejects unsupported shapes with a clear error; it has been running in production for us. PR coming shortly, with the legacy-path schema fix as a separate commit (easy to split out if preferred).

Contributor guide

Open the contributing guide

Research direction

Start with vllm/v1/structured_output/backend_guidance.py, especially serialize_guidance_grammar, and compare its structural-tag handling with backend_xgrammar.py and structural_tag_registry.py. Run the minimal reproduction, then verify that strict auto, required, and named tool choices no longer fail with KeyError and that tool-call arguments remain constrained by their schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.