Idle scheduler loop retains last request's SamplingParams and re-validates it continuously (~30Hz), driving CPU + memory growth with zero active requests

Summary

The experimental OpenAI-compatible server's internal decode/scheduler loop appears to cache the last real request's SamplingParams as a persistent idle-tick template and continuously re-validates it (logging [WARNING][sampling.h:72:SamplingParams] Temperature is 0.0f, but topK is not 1 or topP is not 1.0f...) forever afterward — at a steady ~28-30 Hz — even with zero requests in flight. This is not proportional to real traffic at all: we measured it directly.

Each idle tick does measurable, non-trivial allocator work: RSS grows even during a window with confirmed zero real HTTP requests. Over a live deployment's typical inference lifetime (running as a long-lived service, not a batch job), this compounds into multi-GB memory growth (observed: ~17GB baseline climbing to 25-30GB over 10-15 minutes) until our own health-checking supervisor restarts the container. This repeats indefinitely — confirmed across 69 consecutive restart cycles over 24h with the baseline itself staying flat (no leak across restarts — it's driven entirely by this idle-tick behavior within each process lifetime).

Environment

• TensorRT-Edge-LLM: v0.9.1 (7f061f2, tag v0.9.1)
• Image tag: fire-edgellm-runtime:0.9.1-jp72-dev-trt10162-lockfix
• Hardware: Jetson AGX Orin (unified memory)
• TRT_VERSION: 10.16.1.11
• Model: Qwen3.6-35B-A3B GPTQ-Int4 (prebuilt engine, max_batch_size=1)
• Diffed v0.9.1...v0.10.0: the implicated files (sampling.h, engineExecutor.cpp, llmInferenceRuntime.cpp, slot.cpp) are unchanged in v0.10.0, so this is not fixed there either as far as we can tell from source.

Minimal repro

Against a freshly-started, idle server (confirmed 0 SamplingParams warnings/sec at baseline):

import json, urllib.request
body = json.dumps({
    "model": "qwen3.6-35b-a3b",
    "messages": [{"role": "user", "content": "hi"}],
    "max_tokens": 1,
    "temperature": 0,
}).encode()
req = urllib.request.Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=body, headers={"content-type": "application/json"}, method="POST",
)
urllib.request.urlopen(req, timeout=25)

Before this request: 0 SamplingParams warnings in a 10s window.
After this single request: sustained ~29-31 warnings/sec indefinitely (measured 461/15s, then 440/15s in the following window), with no further requests sent.

Sending the same request but with top_k=1, top_p=1.0 added (fully consistent with temperature=0) does not trigger the ongoing idle-loop state — warning rate stays at 0 before and after.

Isolation proof (zero-traffic memory growth)

With the server already in the "spamming" state from the repro above, we watched RSS in 30s buckets for 90s with the container otherwise completely idle (0 real HTTP requests logged in any bucket):

RSS=24470064kB  warnings/30s=863  http_reqs/30s=0
RSS=24470256kB  warnings/30s=841  http_reqs/30s=0
RSS=24472252kB  warnings/30s=852  http_reqs/30s=0
RSS=24477400kB  warnings/30s=840  http_reqs/30s=0

RSS grew +7.3MB in 90s with zero real requests — driven purely by the idle-tick loop. top -H on the server process shows one dedicated thread sitting at ~10% CPU continuously, consistent with a ~35ms scheduler tick period.

Why this likely affects many deployments, not just us

temperature=0 without also normalizing top_k=1, top_p=1.0 is a very common client pattern for anyone doing deterministic/greedy decoding through an OpenAI-compatible API — most integrations just set temperature=0 and stop there. Any long-lived deployment of the experimental server that ever receives one such request would enter this state and stay in it indefinitely. We suspect this is underreported because most users don't run this server as an always-on service monitored closely enough over hours to notice a slow multi-GB creep, rather than because the trigger condition itself is rare.

Ask

1. Is the idle scheduler intentionally retaining/replaying the last request's SamplingParams for continuous background re-validation while idle? If not, that retention itself looks like the bug.
2. If the retention is intentional (e.g. for keeping a decode slot "warm"), could re-validation be skipped when there's no actual pending work, or could the params be normalized once at ingestion instead of re-checked/logged every tick?
3. Is there a way to disable or throttle this idle re-validation from the server/config side today (we checked config.json and EDGELLM_* env vars — found no relevant knob)?

Happy to provide more logs, the full request/response trace, or test candidate fixes against our Jetson AGX Orin setup.

Trace the idle scheduler loop through sampling.h, engineExecutor.cpp, llmInferenceRuntime.cpp, and slot.cpp, starting with how the last request's SamplingParams are retained after decoding completes. Reproduce the issue with the provided Python request, then verify that idle ticks stop re-validating stale parameters and that warning frequency and RSS remain stable with zero active requests.