PR #683: engine-backed tool parsers (inkling) run skip_special_tokens=true, stripping structural markers before ParserEngine
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Summary
PR #683 registers inkling as an engine-backed tool parser, but on the actual OpenAI serving path the model's structural tool-call markers are stripped before the parser sees them, silently breaking tool-calling for inkling (and any engine-backed parser).
Mechanism (file:line at PR #683 head 945ce0f7)
OpenAIServingChat::MakeToolParserreturnsnullptrfor every engine-backed name (src/vllm/entrypoints/openai/serving_chat.cpp:546-548), soinklingis served byMakeParserEngine/ParserEngine, not the text-seamToolParser.- Upstream
ParserEngine.adjust_requestsetsskip_special_tokens=False; the C++ParserEnginehas noadjust_requestequivalent, andToolParser::adjust_request(e.g.tool_parsers/kimi_k2.cpp:87-93) is never invoked anywhere in the serving path. ChatCompletionRequest.skip_special_tokensdefaultstrue(include/vllm/entrypoints/openai/protocol.h:461) and is forwarded verbatim byto_sampling_params(protocol.cpp:583);serving_chat.cppapplies no override for the engine path.inkling's structural markers (<|message_model|>,<|content_invoke_tool_json|>, …) are tokenizer specials, so a default OpenAI request strips the grammar in the detokenizer beforeParserEngineruns.
Test gap
The 18 new tests feed literal marker text directly to the adapter, so they exercise the adapter but NOT the serving/detokenizer path — they pass while the production path fails.
Fix direction (for the implementer's spec)
Mirror upstream ParserEngine.adjust_request and the existing kimi_k2 precedent: when an engine-backed tool parser is active with tools (has_tools && tool_choice != none), force skip_special_tokens=false. Add a serving-path regression test proving a default OpenAI ChatCompletion request preserves inkling markers end-to-end through the detokenizer — not another adapter unit test.
Provenance
Found in review of PR #683 by the research agent; verified in source and diagnosed by the coordinator. Fix to be done by a fresh implementer + fresh reviewer per AGENTS.md.
Kind: bug
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/vllm/entrypoints/openai/serving_chat.cpp, protocol.h, and protocol.cpp, then compare the upstream ParserEngine.adjust_request behavior with tool_parsers/kimi_k2.cpp. Trace a default ChatCompletion request through the serving and detokenizer paths. Done means engine-backed parsers preserve inkling markers when tools are active and a serving-path regression test verifies this end to end.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100