InternLM / InternLM/lmdeploy

[Bug]version 0.10.1 : gpt-oss-20b and gpt-oss-120b model returns null message.content on API server

Open
#4,049 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

When serving the gpt-oss-20b model using LMDeploy's api_server with tensor parallelism (tp=2), the API successfully accepts requests and returns a 200 OK status. However, the generated content in the response is consistently null, and gen_tokens is also null, even though the usage statistics indicate that tokens were processed. The server logs show no errors, and the GPUs are under load.
he issue is observed when setting up an inference server for the gpt-oss-20b model. The server initializes without any errors, and the API endpoint appears to be functioning correctly. Upon sending a POST request to the /v1/chat/completions endpoint, the server logs a successful 200 OK response.

However, the JSON response payload is problematic. The message.content field, which should contain the generated text from the model, is null. Similarly, the choices[0].gen_tokens field, which should report the number of tokens generated, is also null.

The usage object in the response correctly calculates prompt_tokens, total_tokens, and completion_tokens, which suggests that the request was processed and the model allocated resources for generation, but the actual output text was not produced or returned.

### Reproduction

export TM_ANOMALY_HANDLER=level=2,inf=65504,nan=0
lmdeploy serve api_server /home/ai/models/openai/gpt-oss-20b_ud \
--server-port 8080 \
--tp 2 \
--session-len 131072 \
--max-batch-size 8 \
--cache-max-entry-count 0.8 \
--cache-block-seq-len 32 \
--enable-prefix-caching \
--dtype float16 \
--model-name gpt-oss-20b
[TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 131072.
2025-10-20 10:41:32,571 - lmdeploy - WARNING - turbomind.py:234 - get 14165 model params
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] [AnomalyHandler] level: 2
[TM][WARNING] [AnomalyHandler] +INF -> 65504.000000
[TM][WARNING] [AnomalyHandler] -INF -> -65504.000000
[TM][WARNING] [AnomalyHandler] NaN -> 0.000000
[TM][WARNING] [AnomalyHandler] fallback: 0
[TM][WARNING] [AnomalyHandler] max_batch_size: 8
[TM][WARNING] [AnomalyHandler] vocab_size: 201088
[TM][WARNING] [AnomalyHandler] fallback: 0
[TM][WARNING] [AnomalyHandler] max_batch_size: 8
[TM][WARNING] [AnomalyHandler] vocab_size: 201088
HINT: Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
HINT: Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
HINT: Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
INFO: Started server process [305359]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: 127.0.0.1:58524 - "POST /v1/chat/completions HTTP/1.1" 200 OK

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "gpt-oss-20b",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Hello, who are you?"}
],
"max_tokens": 500,
"temperature": 1
}'
{"id":"1","object":"chat.completion","created":1760928116,"model":"gpt-oss-20b","choices":[{"index":0,"message":{"role":"assistant","content":null,"gen_tokens":null,"reasoning_content":null,"tool_calls":[]},"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":84,"total_tokens":176,"completion_tokens":92}}

### Environment

```Shell
lmdeploy check_env
sys.platform: linux
Python: 3.10.18 (main, Jun 5 2025, 13:14:17) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: Tesla V100-SXM2-32GB
GPU 2: Tesla PG503-216
GPU 3: NVIDIA GeForce RTX 2080 Ti
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
PyTorch: 2.8.0+cu128
PyTorch compiling details: PyTorch built with:
- GCC 13.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 12.8
- NVCC architecture flags: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_100,code=sm_100;-gencode;arch=compute_120,code=sm_120
- CuDNN 91.0.2 (built against CUDA 12.9)
- Built with CuDNN 90.8
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=a1cb3cc05d46d198467bebbb6e8fba50a325d4e7, CUDA_VERSION=12.8, CUDNN_VERSION=9.8.0, CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,

TorchVision: 0.23.0+cu128
LMDeploy: 0.10.1+5c6dac7
transformers: 4.57.1
fastapi: 0.119.0
pydantic: 2.12.3
triton: 3.4.0
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PIX PIX PHB 0-13,28-41 0 N/A
GPU1 PIX X NV6 PHB 0-13,28-41 0 N/A
GPU2 PIX NV6 X PHB 0-13,28-41 0 N/A
GPU3 PHB PHB PHB X 0-13,28-41 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

Which model are you using? gpt-oss-20b
How you installed PyTorch [e.g., pip, conda, source] pip

env:(lmdeploy) root@t-7810:~# export
declare -x CMAKE_CUDA_COMPILER="/usr/local/cuda/bin/nvcc"
declare -x CONDA_DEFAULT_ENV="lmdeploy"
declare -x CONDA_EXE="/home/anaconda3/bin/conda"
declare -x CONDA_PREFIX="/home/anaconda3/envs/lmdeploy"
declare -x CONDA_PREFIX_1="/home/anaconda3"
declare -x CONDA_PREFIX_2="/home/anaconda3/envs/py3.12"
declare -x CONDA_PROMPT_MODIFIER="(lmdeploy) "
declare -x CONDA_PYTHON_EXE="/home/anaconda3/bin/python"
declare -x CONDA_SHLVL="3"
declare -x CUDACXX="/usr/local/cuda/bin/nvcc"
declare -x CUDA_BIN_PATH="/usr/local/cuda/bin"
declare -x CUDA_HOME="/usr/local/cuda"
declare -x CUDA_LIB_DIR="/usr/local/cuda/lib64"
declare -x CUDA_TOOLKIT_ROOT_DIR="/usr/local/cuda"
declare -x DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/0/bus"
declare -x DEBUGINFOD_URLS=""
declare -x HF_ENDPOINT="https://hf-mirror.com"
declare -x HOME="/root"
declare -x LANG="zh_CN.UTF-8"
declare -x LANGUAGE="zh_CN:zh"
declare -x LD_LIBRARY_PATH="/usr/local/cuda/lib64:"
declare -x LESSCLOSE="/usr/bin/lesspipe %s %s"
declare -x LESSOPEN="| /usr/bin/lesspipe %s"
declare -x LOGNAME="root"
declare -x NVM_BIN="/root/.nvm/versions/node/v20.13.1/bin"
declare -x NVM_CD_FLAGS=""
declare -x NVM_DIR="/root/.nvm"
declare -x NVM_INC="/root/.nvm/versions/node/v20.13.1/include/node"

declare -x PATH="/root/.nvm/versions/node/v20.13.1/bin:/usr/local/cuda/bin:/home/soft/go1.22/go/bin:/home/anaconda3/envs/lmdeploy/bin:/home/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
declare -x PWD="/root"
declare -x SHELL="/bin/bash"
declare -x SHLVL="1"
declare -x SSH_TTY="/dev/pts/1"
declare -x TERM="xterm-256color"
declare -x USER="root"
declare -x XDG_DATA_DIRS="/usr/share/gnome:/usr/local/share:/usr/share:/var/lib/snapd/desktop"
declare -x XDG_RUNTIME_DIR="/run/user/0"
declare -x XDG_SESSION_CLASS="user"
declare -x XDG_SESSION_ID="5"
declare -x XDG_SESSION_TYPE="tty"
declare -x _CE_CONDA=""
declare -x _CE_M=""
```

### Error traceback

```Shell
lmdeploy serve api_server /home/ai/models/openai/gpt-oss-20b_ud --server-port 8080 --tp 2 --session-len 131072 --max-batch-size 8 --cache-max-entry-count 0.8 --cache-block-seq-len 32 --enable-prefix-caching --dtype float16 --model-name gpt-oss-20b
[TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 131072.
2025-10-20 10:41:32,571 - lmdeploy - WARNING - turbomind.py:234 - get 14165 model params
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] [SegMgr] prefix caching is enabled
[TM][WARNING] [AnomalyHandler] level: 2
[TM][WARNING] [AnomalyHandler] +INF -> 65504.000000
[TM][WARNING] [AnomalyHandler] -INF -> -65504.000000
[TM][WARNING] [AnomalyHandler] NaN -> 0.000000
[TM][WARNING] [AnomalyHandler] fallback: 0
[TM][WARNING] [AnomalyHandler] max_batch_size: 8
[TM][WARNING] [AnomalyHandler] vocab_size: 201088
[TM][WARNING] [AnomalyHandler] fallback: 0
[TM][WARNING] [AnomalyHandler] max_batch_size: 8
[TM][WARNING] [AnomalyHandler] vocab_size: 201088
HINT: Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
HINT: Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
HINT: Please open http://0.0.0.0:8080 in a browser for detailed api usage!!!
INFO: Started server process [305359]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: 127.0.0.1:58524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.