ROCm / ROCm/FastFlowLM

Server silently serves the resident model for unresolvable model tags, echoing the requested name

Open
#716 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.9k
Forks
152
Avg merge
4h 14m
Merged PRs (30d)
11

Description

Summary

flm serve answers requests for model tags it cannot resolve by running whatever model is
currently loaded
, and echoes the requested tag back in the response model field. Nothing in
the response or the server log indicates the substitution. A benchmark script can therefore collect
timings for a model that never loaded.

Reproduce (10 seconds)
flm serve qwen3:1.7b --host 127.0.0.1 --port 8093

curl -s http://127.0.0.1:8093/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"totally-made-up:99b","messages":[{"role":"user","content":"Name three colors."}],"max_tokens":40}'

Actual — a fully formed completion for a tag that exists nowhere:

{"model":"totally-made-up:99b",
 "choices":[{"message":{"role":"assistant","content":"Here are three colors:\n\n1. Red\n2. Blue\n3. Green"}}],
 "usage":{"decoding_speed_tps":43.27, ...}}

Expected: an error naming the unresolvable tag.

Same on /v1/audio/transcriptions: one loaded Whisper answered to whisper-v3:turbo,
whisper-v3-turbo-FLM and whisper-1 alike, each echoed back verbatim.

Why it matters

The model field is the only signal a client has about which model ran, and it is an echo of the
request rather than a statement of fact. The failure is easiest to hit exactly when a tag should
fail — mid-setup, before model_list.json is correct — which is also when people run their first
benchmark. The result looks completely normal.

Note this is distinct from a registered-but-unloadable tag, which does error correctly.

Suggested fix

Return an error for an unresolvable tag instead of falling through to the resident model. Failing
that, populate the response model field with the model that actually served the request, so the
output is self-describing.

Environment: FLM v1.0.1 / v1.0.2 / v1.0.4, Ryzen AI Max+ 395, NPU fw 1.1.2.65, in-tree
amdxdna 0.7, kernel 7.0.0-31-generic, Ubuntu 26.04.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the HTTP handling path used by flm serve for /v1/chat/completions and /v1/audio/transcriptions, then trace how requested model tags are resolved before inference. Reproduce the fallback with the provided curl request and verify that an unresolvable tag returns an error rather than using the resident model; the response should no longer claim the requested tag ran.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.