llm_inference ignores temperature/top_p/top_k from the input JSON its own help tells you to set them in

Summary

examples/llm/llm_inference.cpp tells the user twice to put sampling parameters in the
input JSON, and then never reads them. temperature, top_p and top_k in the input
file are silently ignored, and generation is always greedy.

The runtime itself supports sampling — request.temperature, shouldUseNonGreedySampling()
and SamplingParams are all wired in cpp/runtime/llmRankRuntime.cpp — so only the example
driver is missing the parse.

Where the instruction is given

examples/llm/llm_inference.cpp:154

// For other sampling parameters (temperature, top_p, top_k), please specify them in the input JSON file

and again in --help, examples/llm/llm_inference.cpp:357

--maxGenerateLength       Override max generate length from input file
                          NOTE: For sampling parameters (temperature, top_p, top_k),
                          please specify them in the input JSON file instead of CLI

Those two comments are the only occurrences of temperature in the file that are not
talker_temperature (the TTS path, which is parsed, at line 935). Nothing assigns
request.temperature from inputData.

Reproduce

Same input file twice, changing only temperature, with a prompt long enough for sampling
to diverge:

{"batch_size": 1, "temperature": 0.3, "max_generate_length": 120,
 "requests": [{"messages": [{"role": "user", "content": [{"type": "text",
   "text": "Describe, in a few sentences, what a search and rescue robot should do when it enters an unfamiliar room."}]}]}]}

temperature 0.0, 3 runs -> 1 distinct output of 3
temperature 0.3, 3 runs -> 1 distinct output of 3

All six generations are byte-identical. With sampling active at 0.3, three runs would not be.

Why it matters beyond the missing feature

Sampling behaviour can differ qualitatively from greedy, so a benchmark taken through this
tool does not describe a deployment that samples. On this model (a 2.44 B Cosmos reasoner,
INT4-AWQ W4A16) at temperature 0.3 with top_p 0.9, open-ended generation enters a
non-terminating repetition loop that greedy decoding does not:

temperature 0.0   finish=stop     7 steps, coherent, terminates on its own
temperature 0.3   finish=length  83 steps, degenerates at step 12 and never recovers

and raising max_generate_length 250 -> 512 -> 2048 extends the loop (24 -> 83 -> 424
steps) without extending the coherent prefix past step 12. Anyone who measured that model
through llm_inference, having set "temperature": 0.3 in the input file as the help text
instructs, would have seen none of it and concluded the model was fine.

This is the same shape as the ENABLE_CUTE_DSL default that silently compiles the INT4
kernels out: a documented knob that does nothing, failing quietly rather than loudly.

Suggested fix

Parse the three fields where max_generate_length and batch_size are already read, and
populate the request. A warning when a recognised sampling key is present but unused would
also have surfaced this immediately.

Environment: DGX B300 (SM103), CUDA 13.0, TensorRT 10.13.2, Edge-LLM v0.10.1 (e8b2952).
Also present on the same file at main, by inspection.

---

Found during the NVIDIA / OpenHackathons / Oracle Open Models Codefest 2026, while
benchmarking an INT4-AWQ Cosmos3-Edge reasoner for an offline-first search-and-rescue
robotics entry (Team UBR Stack). The production target is a Jetson Orin Nano; the B300 is a
bench machine used for evaluation.

Start in examples/llm/llm_inference.cpp where batch_size and max_generate_length are read, then compare the request fields with cpp/runtime/llmRankRuntime.cpp and its sampling support. Verify the input JSON values for temperature, top_p, and top_k reach the request, and rerun the described repeated-generation comparison to confirm sampling is no longer silently ignored.