mlcommons / mlcommons/endpoints

[Bug]: Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" during running performance(online, offline), sglang 0.5.15.post1, 0.5.18

Open
#497 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

status: needs-triage type: bug
Dominant language
Python
Stars
21
Forks
28
Avg merge
3d 17h
Merged PRs (30d)
13

Description

Bug Description

Warning/Error "parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)" appears when running a performance experiments for sglang 0.5.15.post1, sglang 0.5.18.

Steps to Reproduce
  1. Run a server
docker run --gpus all --shm-size 32g -p 30000:30000 --ipc=host   -v "$HF_CACHE:/root/.cache/huggingface"   -e "HF_TOKEN=$HF_TOKEN"   lmsysorg/sglang:v0.5.18   python3 -m sglang.launch_server   --model-path deepseek-ai/DeepSeek-R1   --trust-remote-code   --tp 8   --context-length 32768   --mem-fraction-static 0.65   --disable-radix-cache   --max-running-requests 2048   --cuda-graph-max-bs-decode 2048   --cuda-graph-bs 128 256 512 1024 2048   --chunked-prefill-size 65536   --max-prefill-tokens 65536   --attention-backend flashinfer   --stream-interval 10   --decode-log-interval 1   --host 0.0.0.0   --port 30000   2>&1 | tee -a "$RUN_DIR/run.log"
  1. Run a performance experiment
uv run inference-endpoint benchmark from-config --config examples/07_DeepSeekR1_Example/performance_h200_online_max_concurrency_test.yaml
...
2026-09-03 08:56:52,162 - inference_endpoint.endpoint_client.http_client - INFO - EndpointClient initialized with num_workers=4, endpoints=['http://localhost:30000/generate'], adapter=SGLangGen
erateAdapter, accumulator=SGLangSSEAccumulator, transport=zmq
2026-09-03 08:56:52,162 - inference_endpoint.load_generator.session - INFO - Starting phase: warmup (warmup)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W2/3618620] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
...

Solution:

diff --git a/src/inference_endpoint/endpoint_client/adapter_protocol.py b/src/inference_endpoint/endpoint_client/adapter_protocol.py
index a96faaa..1f58c59 100644
--- a/src/inference_endpoint/endpoint_client/adapter_protocol.py
+++ b/src/inference_endpoint/endpoint_client/adapter_protocol.py
@@ -127,12 +127,11 @@ class HttpRequestAdapter(ABC):
         """
         json_docs = cls.SSE_DATA_PATTERN.findall(buffer[:end_pos])
         parsed: list[Any] = []
-        # Note: if one frame is malformed, remaining frames are skipped
-        try:
-            for json_doc in json_docs:
+        for json_doc in json_docs:
+            try:
                 content = cls.decode_sse_message(json_doc)
                 if content is not None:
                     parsed.append(content)
-        except (msgspec.DecodeError, msgspec.ValidationError) as exc:
-            logger.warning("skipping malformed SSE batch (%s)", type(exc).__name__)
+            except (msgspec.DecodeError, msgspec.ValidationError) as exc:
+                logger.warning("skipping malformed SSE frame (%s)", type(exc).__name__)
         return parsed
diff --git a/src/inference_endpoint/sglang/adapter.py b/src/inference_endpoint/sglang/adapter.py
index a6614fa..c19e3d2 100644
--- a/src/inference_endpoint/sglang/adapter.py
+++ b/src/inference_endpoint/sglang/adapter.py
@@ -122,7 +122,7 @@ class SGLangGenerateAdapter(HttpRequestAdapter):
         # the output_tokens is the delta from the previous chunk
         token_delta = resp.output_ids
 
-        has_retractions = resp.meta_info.total_retractions > 0
+        has_retractions = resp.meta_info.num_retractions > 0
         return SGLangSSEDelta(
             text=total_text,
             token_delta=token_delta,
diff --git a/src/inference_endpoint/sglang/types.py b/src/inference_endpoint/sglang/types.py
index 9e75b3a..89f71ee 100644
--- a/src/inference_endpoint/sglang/types.py
+++ b/src/inference_endpoint/sglang/types.py
@@ -52,13 +52,13 @@ class SGLangGenerateRequest(
 
 class MetaInfo(msgspec.Struct, frozen=True, kw_only=True, omit_defaults=True, gc=False):  # type: ignore[call-arg]
     id: str
-    finish_reason: dict[str, Any]
+    finish_reason: dict[str, Any] | None
     prompt_tokens: int
     weight_version: str
-    total_retractions: int
+    num_retractions: int
     completion_tokens: int
     cached_tokens: int
-    e2e_latency: float
+    e2e_latency: float | None = None
 
 
 class SGLangGenerateResponse(
diff --git a/tests/performance/sglang/test_sglang_adapter.py b/tests/performance/sglang/test_sglang_adapter.py
index f614cc0..3e0fa90 100644
--- a/tests/performance/sglang/test_sglang_adapter.py
+++ b/tests/performance/sglang/test_sglang_adapter.py
@@ -64,7 +64,7 @@ def make_response_bytes(text: str, n_tokens: int) -> bytes:
                 "finish_reason": {"type": "stop"},
                 "prompt_tokens": 10,
                 "weight_version": "v1",
-                "total_retractions": 0,
+                "num_retractions": 0,
                 "completion_tokens": n_tokens,
                 "cached_tokens": 0,
                 "e2e_latency": 0.1,
diff --git a/tests/unit/endpoint_client/test_adapter_protocol.py b/tests/unit/endpoint_client/test_adapter_protocol.py
index 591dc42..3a177a6 100644
--- a/tests/unit/endpoint_client/test_adapter_protocol.py
+++ b/tests/unit/endpoint_client/test_adapter_protocol.py
@@ -49,7 +49,7 @@ class _SimpleAdapter(HttpRequestAdapter):
 @pytest.mark.unit
 def test_parse_sse_chunk_skips_bad_frame_and_keeps_valid():
     """A malformed SSE frame is skipped; surrounding valid frames are preserved."""
-    buffer = b'data: {"ok":1}\n\ndata: BAD\n\ndata: {"ok":1}\n\n'
+    buffer = b'data: {"ok":1}\n\ndata: {"bad":}\n\ndata: {"ok":1}\n\n'
     end_pos = len(buffer)
     result = _SimpleAdapter.parse_sse_chunk(buffer, end_pos)
     assert result == [{"x": 1}, {"x": 1}]
Environment
git log -1
commit 73981a84fbb8c2dbd761686b3d002ca7f434fe72 (HEAD -> main, origin/main, origin/HEAD)
Relevant Logs
Short run:

uv run inference-endpoint benchmark from-config \
> --config examples/07_DeepSeekR1_Example/performance_h200_online_max_concurrency_test.yaml
2026-09-03 08:56:39,955 - inference_endpoint.endpoint_client.cpu_affinity - INFO - CPU affinity: 128 online CPUs available to process
2026-09-03 08:56:39,960 - inference_endpoint.endpoint_client.cpu_affinity - INFO - CPU affinity: 64 physical cores across 2 NUMA nodes, requesting 5 for loadgen, 4 workers
2026-09-03 08:56:39,988 - inference_endpoint.endpoint_client.cpu_affinity - INFO - LoadGen pinned to 10 CPUs (5 physical cores)
2026-09-03 08:56:40,169 - httpx - INFO - HTTP Request: GET https://huggingface.co/api/models/deepseek-ai/DeepSeek-R1 "HTTP/1.1 200 OK"
2026-09-03 08:56:40,170 - inference_endpoint.commands.benchmark.execute - INFO - Tokenizer available for model: deepseek-ai/DeepSeek-R1
2026-09-03 08:56:40,170 - inference_endpoint.commands.benchmark.execute - INFO - Streaming: enabled (on)
2026-09-03 08:56:40,170 - inference_endpoint.commands.benchmark.execute - INFO - No separate accuracy datasets provided
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - INFO - Loaded 4388 samples
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - WARNING - Warmup is enabled without salt (--warmup-salt): warmup prompts are issued verbatim, so the server may serve t
he measured phase from a warm KV/prefix cache and understate latency. Enable --warmup-salt on text-'prompt' datasets to bust the cache.
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - INFO - Mode: TestMode.PERF, Target QPS: None, Responses: False
2026-09-03 08:56:40,901 - inference_endpoint.commands.benchmark.execute - INFO - Expected samples: 12
deepseek-ai/DeepSeek-R1 (Streaming: True):   0%|                                                                                                                          | 0/12 [00:00<?, ?it/s]
2026-09-03 08:56:40,910 - inference_endpoint.async_utils.transport.zmq.pubsub - INFO - Publisher bound to ipc:///dev/shm/zmq_cjpkdbk5/ev_pub_abfd9bee
2026-09-03 08:56:40,911 - inference_endpoint.async_utils.transport.zmq.pubsub - INFO - Subscriber connected to ipc:///dev/shm/zmq_cjpkdbk5/metrics_pub_b8200cce
2026-09-03 08:56:40,911 - inference_endpoint.async_utils.services.launcher - INFO - Launching service: inference_endpoint.async_utils.services.metrics_aggregator (id=0)
2026-09-03 08:56:40,911 - inference_endpoint.async_utils.services.launcher - INFO - Launching service: inference_endpoint.async_utils.services.event_logger (id=1)
2026-09-03 08:56:41,342 - inference_endpoint.async_utils.transport.zmq.pubsub - INFO - Subscriber connected to ipc:///dev/shm/zmq_cjpkdbk5/ev_pub_abfd9bee
2026-09-03 08:56:42,767 - httpx - INFO - HTTP Request: HEAD https://huggingface.co/deepseek-ai/DeepSeek-R1/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
...
2026-09-03 08:56:52,162 - inference_endpoint.endpoint_client.http_client - INFO - EndpointClient initialized with num_workers=4, endpoints=['http://localhost:30000/generate'], adapter=SGLangGen
erateAdapter, accumulator=SGLangSSEAccumulator, transport=zmq
2026-09-03 08:56:52,162 - inference_endpoint.load_generator.session - INFO - Starting phase: warmup (warmup)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W2/3618620] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W0/3618618] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
2026-09-03 08:56:52,743 - inference_endpoint.endpoint_client.adapter_protocol[W1/3618619] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
...
2026-09-03 09:02:25,578 - inference_endpoint.endpoint_client.adapter_protocol[W3/3618621] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
2026-09-03 09:02:25,669 - inference_endpoint.endpoint_client.adapter_protocol[W3/3618621] - parse_sse_chunk - WARNING - skipping malformed SSE batch (ValidationError)
deepseek-ai/DeepSeek-R1 (Streaming: True): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [05:44<00:00, 28.73s/it]2026-09-03 09:02:25,670 - inference_endpoint.load_generator.session - INFO - Phase performance complete: 12 samples issued
2026-09-03 09:02:25,670 - inference_endpoint.commands.benchmark.pipeline - INFO - Closing publisher (buffer=0, pending=0)...
2026-09-03 09:02:25,670 - inference_endpoint.commands.benchmark.pipeline - INFO - Waiting for services to finish processing...
2026-09-03 09:02:25,671 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - ENDED event received, shutting down aggregator
2026-09-03 09:02:25,671 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - Draining 0 pending tokenizations...
2026-09-03 09:02:25,671 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - Tokenizations fully drained (n_pending_tasks=0)
2026-09-03 09:02:25,675 - inference_endpoint.async_utils.services.metrics_aggregator.registry - INFO - sample_latency_ns early-stopping detail (confidence 0.99): p99.9: estimate=None empirical=96469560463 n=12 min_queries=6636 discarded=0; p99.0: estimate=None empirical=96469560463 n=12 min_queries=662 discarded=0; p97.0: estimate=None empirical=96469560463 n=12 min_queries=219 discarded=0; p95.0: estimate=None empirical=96469560463 n=12 min_queries=130 discarded=0; p90.0: estimate=None empirical=82004097015 n=12 min_queries=64 discarded=0; p80.0: estimate=None empirical=72922931653 n=12 min_queries=31 discarded=0; p75.0: estimate=None empirical=72922931653 n=12 min_queries=24 discarded=0; p50.0: estimate=164572395605 empirical=48464537381 n=12 min_queries=11 discarded=0
2026-09-03 09:02:25,676 - inference_endpoint.async_utils.services.metrics_aggregator.registry - INFO - ttft_ns early-stopping detail (confidence 0.99): p99.9: estimate=None empirical=None n=0 min_queries=6636 discarded=0; p99.0: estimate=None empirical=None n=0 min_queries=662 discarded=0; p97.0: estimate=None empirical=None n=0 min_queries=219 discarded=0; p95.0: estimate=None empirical=None n=0 min_queries=130 discarded=0; p90.0: estimate=None empirical=None n=0 min_queries=64 discarded=0; p80.0: estimate=None empirical=None n=0 min_queries=31 discarded=0; p75.0: estimate=None empirical=None n=0 min_queries=24 discarded=0; p50.0: estimate=None empirical=None n=0 min_queries=11 discarded=0
2026-09-03 09:02:25,676 - inference_endpoint.async_utils.services.metrics_aggregator.registry - INFO - tpot_ns early-stopping detail (confidence 0.99): p99.9: estimate=None empirical=None n=0 min_queries=6636 discarded=0; p99.0: estimate=None empirical=None n=0 min_queries=662 discarded=0; p97.0: estimate=None empirical=None n=0 min_queries=219 discarded=0; p95.0: estimate=None empirical=None n=0 min_queries=130 discarded=0; p90.0: estimate=None empirical=None n=0 min_queries=64 discarded=0; p80.0: estimate=None empirical=None n=0 min_queries=31 discarded=0; p75.0: estimate=None empirical=None n=0 min_queries=24 discarded=0; p50.0: estimate=None empirical=None n=0 min_queries=11 discarded=0
2026-09-03 09:02:25,679 - inference_endpoint.async_utils.services.metrics_aggregator.aggregator - INFO - Aggregator finalized: 36 total records processed
2026-09-03 09:02:25,804 - inference_endpoint.commands.benchmark.pipeline - INFO - Built report from final_snapshot.json
deepseek-ai/DeepSeek-R1 (Streaming: True): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [05:44<00:00, 28.74s/it]
2026-09-03 09:02:25,815 - inference_endpoint.endpoint_client.http_client - INFO - [e54f55c3] Shutting down...
2026-09-03 09:02:26,316 - inference_endpoint.endpoint_client.http_client - INFO - [e54f55c3] Shutdown complete.
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - ----------------- Summary -----------------
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Version: 0.1.0
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Git SHA: 73981a8
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Run config:
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   load_pattern: type=concurrency, target_qps=None, target_concurrency=4
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   warmup: enabled=True, n_requests=8, salt=False, drain=True, warmup_random_seed=42
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   scheduler_random_seed: 42
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO -   dataloader_random_seed: 42
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples issued: 12
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples succeeded: 12
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples failed: 0
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Total samples dropped: 0
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - Duration: 261.04 seconds
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - QPS: 0.05
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - TPS: N/A
2026-09-03 09:02:26,320 - inference_endpoint.commands.benchmark.execute - INFO - ----------------- End of Summary -----------------
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Report written to results/deepseek_r1_h200/online_max_concurrency_4_20260903_sglang_0_5_18/report.txt
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Completed in 261.0s
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Results: 12/12 successful
2026-09-03 09:02:26,322 - inference_endpoint.commands.benchmark.execute - INFO - Estimated QPS: 0.0
2026-09-03 09:02:26,324 - inference_endpoint.commands.benchmark.execute - INFO - Partial results saved to results/deepseek_r1_h200/online_max_concurrency_4_20260903_sglang_0_5_18
Before submitting
  • I searched existing issues and found no duplicates

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/inference_endpoint/endpoint_client/adapter_protocol.py and its parser test in tests/unit/endpoint_client/test_adapter_protocol.py to understand how malformed SSE frames are handled. Then inspect src/inference_endpoint/sglang/adapter.py and types.py, along with tests/performance/sglang/test_sglang_adapter.py, for the SGLang response fields. Done means valid frames are retained, current SGLang responses validate, and the listed tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.