mudler / mudler/vllm.cpp

PR #683: engine-backed tool parsers (inkling) run skip_special_tokens=true, stripping structural markers before ParserEngine

Open
#695 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Summary

PR #683 registers inkling as an engine-backed tool parser, but on the actual OpenAI serving path the model's structural tool-call markers are stripped before the parser sees them, silently breaking tool-calling for inkling (and any engine-backed parser).

Mechanism (file:line at PR #683 head 945ce0f7)

  • OpenAIServingChat::MakeToolParser returns nullptr for every engine-backed name (src/vllm/entrypoints/openai/serving_chat.cpp:546-548), so inkling is served by MakeParserEngine / ParserEngine, not the text-seam ToolParser.
  • Upstream ParserEngine.adjust_request sets skip_special_tokens=False; the C++ ParserEngine has no adjust_request equivalent, and ToolParser::adjust_request (e.g. tool_parsers/kimi_k2.cpp:87-93) is never invoked anywhere in the serving path.
  • ChatCompletionRequest.skip_special_tokens defaults true (include/vllm/entrypoints/openai/protocol.h:461) and is forwarded verbatim by to_sampling_params (protocol.cpp:583); serving_chat.cpp applies no override for the engine path.
  • inkling's structural markers (<|message_model|>, <|content_invoke_tool_json|>, …) are tokenizer specials, so a default OpenAI request strips the grammar in the detokenizer before ParserEngine runs.

Test gap

The 18 new tests feed literal marker text directly to the adapter, so they exercise the adapter but NOT the serving/detokenizer path — they pass while the production path fails.

Fix direction (for the implementer's spec)

Mirror upstream ParserEngine.adjust_request and the existing kimi_k2 precedent: when an engine-backed tool parser is active with tools (has_tools && tool_choice != none), force skip_special_tokens=false. Add a serving-path regression test proving a default OpenAI ChatCompletion request preserves inkling markers end-to-end through the detokenizer — not another adapter unit test.

Provenance

Found in review of PR #683 by the research agent; verified in source and diagnosed by the coordinator. Fix to be done by a fresh implementer + fresh reviewer per AGENTS.md.

Kind: bug

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/vllm/entrypoints/openai/serving_chat.cpp, protocol.h, and protocol.cpp, then compare the upstream ParserEngine.adjust_request behavior with tool_parsers/kimi_k2.cpp. Trace a default ChatCompletion request through the serving and detokenizer paths. Done means engine-backed parsers preserve inkling markers when tools are active and a serving-path regression test verifies this end to end.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.