NVIDIA-NeMo / NVIDIA-NeMo/Export-Deploy
Server returns empty answer when too many tokens requested
Open
@athitten is already working on this.
Since Apr 13, 2026.
bug
- Dominant language
- Python
- Stars
- 42
- Forks
- 18
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 8
Description
Describe the bug
When the "max_tokens" in the payload is higher than --inference_max_seq_length passed to the server (in my case: 8192 vs 4096) the server responds with empty assistant message.
Steps/Code to reproduce bug
Deployment snippet (Eos cluster):
python \
/opt/Export-Deploy/scripts/deploy/nlp/deploy_ray_inframework.py \
--megatron_checkpoint /lustre/fsw/coreai_dlalgo_ci/nemo_export_deploy_eval_checkpoints/mbridge/meta-llama/Llama-3.1-8B-Instruct/iter_0000000/ \
--model_id megatron_model \
--port 8886 \
--host 0.0.0.0 \
--num_gpus 8 \
--tensor_model_parallel_size 1 \
--pipeline_model_parallel_size 1 \
--expert_model_parallel_size 1 \
--max_batch_size 2 \
--num_replicas 8 \
--inference_max_seq_length 4096 \
--runtime_env '{"py_executable": "/opt/venv/bin/python"}' &
Then to send a request to the model:
import requests
model_name="megatron_model"
endpoint_url="http://0.0.0.0:8886/v1/chat/completions"
payload = {"model": model_name, "max_tokens": 8192, "top_p": 0.9999999, "temperature": 1e-07, "messages": [{"role": "user", "content": "## Instruction:\n\nPlease answer this question by first reasoning and then selecting the correct choice.\nPresent your reasoning and solution in the following json format.\nPlease show your choice in the `answer` field with only the choice letter, e.g.,`\"answer\": \"C\"`.\n\n```json\n{\n \"reasoning\": \"___\",\n \"answer\": \"___\"\n}\n```\n\n## Question:\n\nWhich of the following is a disorder characterized by uncontrollable episodes of falling asleep during the day?\n\n## Choices:\n\n- (A) Dyslexia\n- (B) Epilepsy\n- (C) Hydrocephalus\n- (D) Narcolepsy\n\n## Answer:"}]}
response = requests.post(endpoint_url, json=payload)
response.json()
Expected behavior
The server should respond with descriptive error
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.