[Bug] Wrong inference using docker container's sglang server with Qwen3-VL models

Bug Description

I am using the sglang inside the slime docker to serve a Qwen3-VL-4B-instruct model with a simple vqa task. the served model cannot correctly see the image, and may output ???? or incorrect analysis.

Steps to Reproduce

1. serve the Qwen3-VL-4B-instruct model with

   python -m sglang.launch_server --config examples/visual_reasoning_zero/configs/serve_qwen3_vl_4b.yaml
   

   with config

   model-path: ./pretrained_models/Qwen3-VL-4B-Instruct
   host: localhost
   port: 30000
   data-parallel-size: 8
   mem-fraction-static: 0.85
   

2. run with these two examples (openai compatible and sglang native apis)

# try with openai api
import requests
import base64

img_resp = requests.get(
    "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
)
b64 = base64.b64encode(img_resp.content).decode()

messages: List[Dict[str, Any]] = [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{b64}"
                    },
                },
            ],
        }
    ]

client.chat.completions.create(**params)

# try with sglang native api
from transformers import AutoTokenizer, AutoProcessor
from httpx import post

model_path = "/root/visual-thinking-zero/pretrained_models/Qwen3-VL-4B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_path)
processor = AutoProcessor.from_pretrained(model_path)
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_dict=True)
sampling_params = {
    "max_new_tokens": 1024,
    "temperature": 1.0,
    "repetition_penalty": 1.0,
}
payload = {
    "input_ids": inputs["input_ids"],
    "sampling_params": sampling_params,
    "return_logprob": True,
}
output = post(url="http://127.0.0.1:30000/generate", json=payload)

Expected Behavior

the image is a man on a yellow taxi

Actual Behavior

1. OpenAI API case:

ChatCompletion(id='b5af35514aa24bf7bafb23881a34f499', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='The image shows a series of three identical, overlapping, and distorted square shapes. These squares are arranged in a grid-like pattern, with each square appearing to be slightly misaligned or rotated relative to the others, creating a sense of motion or distortion. The squares are filled with a solid, uniform color, and there are no visible borders or outlines around them. The overall effect is one of a fragmented or abstract representation of a grid or pattern.', refusal=None, role='assistant', annotations=None, audio=None, function_call=None, tool_calls=None, reasoning_content=None), matched_stop=151645)], created=1776762502, model='Qwen/Qwen3-VL-4B-Instruct', object='chat.completion', service_tier=None, system_fingerprint=None, usage=CompletionUsage(completion_tokens=90, prompt_tokens=232, total_tokens=322, completion_tokens_details=None, prompt_tokens_details=None, reasoning_tokens=0), metadata={'weight_version': 'default'})

2. SGlang native

b'[{"text":"This image shows a pair of **black suede ankle boots**. They feature a sleek, minimalist design with the following details:\\n\\n- **Material**: The upper is made of soft black suede.\\n- **Style**: Ankle boots with a low, slightly\\nassistant\\nThis image shows a very close-up, slightly blurry view of a person\xe2\x80\x99s hand holding a dark-colored smartphone. The hand is gripping the phone, and the screen is turned on, displaying a grid of colorful app icons on a dark background.\\n\\nVisible details squared-off heel.\\n- **Design**: They have a clean, simple silhouette \xe2\x80\x94 no visible laces or buckles, suggesting they may be slip-on or have hidden closure.\\n- **Soles**: The soles are dark, likely black or dark:\\n- The phone has a large screen with rounded corners.\\n- The icons include a calculator (blue), a stylus (pink), and several others in various colors.\\n- The icons are arranged in a grid layout, typical of an Android or iOS brown, with a slight platform or heel for added height.\\n\\nThe boots are photographed on a light-colored, possibly white, background, which helps highlight their sleek, modern look.","output_ids":[1986,2168,4933,264,6716,315,3070,11453,96635,38348,25236,334,13,2379,4565,264,47394,11,71670,2884,448,279,2701,3565,1447,12,3070,13415,95518,576,8416,374,1865,315,8413,3691,96635,624,12,3070,2323,95518,1527,23089,25236,448,264,3347,11,10078,1986,2168,4933,264,1602,3265,5239,11,10078,99055,1651,315,264,1697,748,1424,9963,264,6319,57722,21511,13,576,1424,374,80191,279,4540,11,323,279,4171,374,6519,389,11,27940,264,5827,315,33866,906,23765,389,264,6319,4004,382,5715,3565,52263,12462,34328,624,12,3070,20470,95518,2379,614,264,4240,11,4285,56727,1959,902,9434,326,2434,476,22012,642,11,22561,807,1231,387,21134,10326,476,614,8177,21955,624,12,3070,4416,642,95518,576,773,642,525,6319,11,4363,3691,476,6319,510,12,576,4540,702,264,3460,4171,448,17976,23462,624,12,576,23765,2924,264,29952,320,12203,701,264,48204,355,320,63249,701,323,3807,3800,304,5257,7987,624,12,576,23765,525,27802,304,264,5827,6789,11,14260,315,458,8514,476,16048,13876,11,448,264,8112,5339,476,34328,369,3694,2608,382,785,25236,525,56203,389,264,3100,57722,11,10767,4158,11,4004,11,892,8609,11167,862,47394,11,6481,1401,13,151645],"meta_info":{"id":"c7058fa8f1ee417a8d58a4619e2099a2","finish_reason":{"type":"stop","matched":151645},"prompt_tokens":17,"weight_version":"default","total_retractions":0,"input_token_logprobs":[[null,198,null],[null,198,null]],"output_token_logprobs":[[-0.3172343671321869,1986,null],[-0.0067281657829880714,2168,null],[-0.5503875613212585,4933,null],[-0.2438332736492157,264,null],[-4.278907299041748,6716,null],[-1.7881409348774469e-06,315,null],[-1.1939020156860352,3070,null],[-1.7226617336273193,11453,null],[-5.078724384307861,96635,null],[-0.7213377952575684,38348,null],[-0.0025868467055261135,25236,null],[-0.0049537126906216145,334,null],[-2.2351367473602295,13,null],[-0.4621894955635071,2379,null],[-0.7570490837097168,4565,null],[-1.653856635093689,264,null],[-2.457170248031616,47394,null],[-0.008297620341181755,11,null],[-0.253244549036026,71670,null],[-2.3246082491823472e-05,2884,null],[-0.0026482809334993362,448,null],[-1.7992727756500244,279,null],[-3.576279254957626e-07,2701,null],[-0.36204200983047485,3565,null],[-0.023252340033650398,1447,null],[-0.0012035457184538245,12,null],[-0.008666650392115116,3070,null],[-0.01996399275958538,13415,null],[-0.04873733967542648,95518,null],[-0.7606918811798096,576,null],[-0.4474859833717346,8416,null],[-0.07362592220306396,374,null],[-0.004648805130273104,1865,null],[-0.003178636310622096,315,null],[-1.7659904956817627,8413,null],[-0.10731218010187149,3691,null],[-5.960466182841628e-07,96635,null],[-0.031185204163193703,624,null],[-2.3841860752327193e-07,12,null],[-2.3841860752327193e-07,3070,null],[-0.3425350785255432,2323,null],[-2.014657366089523e-05,95518,null],[-0.07351140677928925,1527,null],[-3.695494797284482e-06,23089,null],[-1.1614398956298828,25236,null],[-0.018433934077620506,448,null],[-9.63257480179891e-05,264,null],[-0.48318928480148315,3347,null],[-0.3351258933544159,11,null],[-0.941139280796051,10078,null],[-0.2561345100402832,1986,null],[-0.0076223099604249,2168,null],[-0.5006101131439209,4933,null],[-0.2456672489643097,264,null],[-5.033129692077637,1602,null],[-3.2533833980560303,3265,null],[-0.0002908533497247845,5239,null],[-0.22394651174545288,11,null],[-0.6113002300262451,10078,null],[-0.1101033166050911,99055,null],[-0.38164374232292175,1651,null],[-0.0002458993985783309,315,null],[-0.23740464448928833,264,null],[-1.21466863155365,1697,null],[-0.5781996250152588,748,null],[-1.1190402507781982,1424,null],[-0.2640804946422577,9963,null],[-0.016841834411025047,264,null],[-3.8164939880371094,6319,null],[-1.1095398664474487,57722,null],[-1.3000174760818481,21511,null],[-0.15547862648963928,13,null],[-0.001463530003093183,576,null],[-1.088762879371643,1424,null],[-0.0690624788403511,374,null],[-2.7284529209136963,80191,null],[-3.171017306158319e-05,279,null],[-0.079045370221138,4540,null],[-1.4302279949188232,11,null],[-0.6583555340766907,323,null],[-0.27447763085365295,279,null],[-0.38381871581077576,4171,null],[-0.2764197289943695,374,null],[-2.7326266765594482,6519,null],[-0.16094756126403809,389,null],[-0.031073909252882004,11,null],[-0.03381914272904396,27940,null],[-0.2677406370639801,264,null],[-4.20502233505249,5827,null],[-0.006066266912966967,315,null],[-0.21860475838184357,33866,null],[-0.00996068213135004,906,null],[-7.391003236989491e-06,23765,null],[-2.574622631072998,389,null],[-0.0349297970533371,264,null],[-1.178346872329712,6319,null],[-0.00354793481528759,4004,null],[-2.879045009613037,382,null],[-3.495227575302124,5715,null],[-0.11766950786113739,3565,null],[-4.002889633178711,52263,null],[-0.7096423506736755,12462,null],[-0.062044594436883926,34328,null],[-0.1535133421421051,624,null],[-1.1920930376163597e-07,12,null],[-1.0728841743912199e-06,3070,null],[-2.1972172260284424,20470,null],[-0.17462557554244995,95518,null],[-1.9760651588439941,2379,null],[-0.0025002595502883196,614,null],[-0.00457227835431695,264,null],[-1.5052306652069092,4240,null],[-0.01698017306625843,11,null],[-2.1441540718078613,4285,null],[-0.06309214234352112,56727,null],[-6.190055847167969,1959,null],[-0.006921947002410889,902,null],[-0.14577381312847137,9434,null],[-0.13115589320659637,326,null],[-0.4741363227367401,2434,null],[-0.977140486240387,476,null],[-2.6074442863464355,22012,null],[-1.0132840543519706e-05,642,null],[-0.21232692897319794,11,null],[-0.2945619821548462,22561,null],[-0.5864546895027161,807,null],[-0.8757941722869873,1231,null],[-0.03104882314801216,387,null],[-0.1601702868938446,21134,null],[-0.023333359509706497,10326,null],[-0.003555890405550599,476,null],[-0.045570164918899536,614,null],[-0.4434000849723816,8177,null],[-2.408346176147461,21955,null],[-0.14375075697898865,624,null],[-9.536747711536009e-07,12,null],[-6.7949526965094265e-06,3070,null],[-1.483168601989746,4416,null],[-0.003608530852943659,642,null],[-0.00014067685697227716,95518,null],[-0.2052305042743683,576,null],[-0.0760122612118721,773,null],[-3.576279254957626e-07,642,null],[-0.8269513845443726,525,null],[-0.5770734548568726,6319,null],[-0.42687904834747314,11,null],[-0.32391276955604553,4363,null],[-0.8857966661453247,3691,null],[-0.32591012120246887,476,null],[-0.017154909670352936,6319,null],[-0.987492561340332,510,null],[-0.004760971292853355,12,null],[-0.28561916947364807,576,null],[-0.10511969774961472,4540,null],[-0.631561815738678,702,null],[-0.0037778958212584257,264,null],[-2.1235134601593018,3460,null],[-0.8044676780700684,4171,null],[-0.3513725697994232,448,null],[-0.4872593283653259,17976,null],[-0.005261829122900963,23462,null],[-0.24893061816692352,624,null],[-6.437322554120328e-06,12,null],[-0.15981349349021912,576,null],[-0.9553184509277344,23765,null],[-1.7306373119354248,2924,null],[-1.2806380987167358,264,null],[-5.384982585906982,29952,null],[-3.0569543838500977,320,null],[-1.4135761260986328,12203,null],[-0.02883826196193695,701,null],[-0.020012397319078445,264,null],[-5.9131999015808105,48204,null],[-0.0005541787249967456,355,null],[-0.9198334813117981,320,null],[-2.321012020111084,63249,null],[-0.09148094803094864,701,null],[-1.631002426147461,323,null],[-1.4519212245941162,3807,null],[-0.8417963981628418,3800,null],[-1.8730616569519043,304,null],[-0.294734925031662,5257,null],[-0.021768657490611076,7987,null],[-0.5870949625968933,624,null],[-1.78815535036847e-05,12,null],[-0.14307387173175812,576,null],[-2.8407108783721924,23765,null],[-0.21568308770656586,525,null],[-0.028848016634583473,27802,null],[-0.015711119398474693,304,null],[-0.17787009477615356,264,null],[-0.7063161134719849,5827,null],[-1.9448175430297852,6789,null],[-0.848034679889679,11,null],[-0.48610740900039673,14260,null],[-0.030543917790055275,315,null],[-1.237253189086914,458,null],[-0.2665930688381195,8514,null],[-0.13853727281093597,476,null],[-0.0015727125573903322,16048,null],[-0.013628906570374966,13876,null],[-0.005958871450275183,11,null],[-0.8890942335128784,448,null],[-0.004175068344920874,264,null],[-0.7169181108474731,8112,null],[-2.027602195739746,5339,null],[-0.16887228190898895,476,null],[-1.5399839878082275,34328,null],[-0.9444552659988403,369,null],[-0.49335217475891113,3694,null],[-0.01589617319405079,2608,null],[-2.5479440689086914,382,null],[-0.17158401012420654,785,null],[-0.012159241363406181,25236,null],[-0.016328541561961174,525,null],[-0.48102307319641113,56203,null],[-1.2754634618759155,389,null],[-4.6015844418434426e-05,264,null],[-3.632523536682129,3100,null],[-0.3491091728210449,57722,null],[-0.1973721832036972,11,null],[-0.3596988320350647,10767,null],[-1.0534812211990356,4158,null],[-0.5814106464385986,11,null],[-0.5370119214057922,4004,null],[-0.08331213891506195,11,null],[-0.11283621191978455,892,null],[-3.8465137481689453,8609,null],[-1.1721688508987427,11167,null],[-0.01106080412864685,862,null],[-3.5121073722839355,47394,null],[-0.4287527799606323,11,null],[-0.13063770532608032,6481,null],[-0.8937095403671265,1401,null],[-0.05450638383626938,13,null],[-3.0754308700561523,151645,null]],"completion_tokens":136,"cached_tokens":10,"cached_tokens_details":null,"e2e_latency":1.0211520195007324,"response_sent_to_client_ts":1776763819.8888693}}]'

Environment

• slime version: 0.2.3
• Python version: 3.12.3
• PyTorch version: 2.9.1+cu129
• CUDA/ROCm version: 13.0
• GPU type and count: A100x8
• OS: linux/official docker container
• SGLang version (if relevant): 0.5.9
• Megatron-LM version (if relevant): 0.16.0rc0

Logs

Additional Context

Maybe the cause is about sglang is patched up to 0.5.9, while there is an issue about Qwen3 behaviors at this sglang issue

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and my issue is not addressed there.
• I have searched for existing issues and this is not a duplicate.
• I have provided a minimal, reproducible example.

Start by reproducing the Qwen3-VL-4B-Instruct request with examples/visual_reasoning_zero/configs/serve_qwen3_vl_4b.yaml, then compare the OpenAI-compatible and native /generate paths. Check how each path constructs and forwards multimodal inputs; the issue is done when both APIs receive the image correctly and identify the man on a yellow taxi.