vllm-project / vllm-project/aibrix

SGLang PD: prefill failure is silent and leaves the decode request hanging on bootstrap

Open
#2,289 3 comments 0 reactions 0 assignees View on GitHub
area/disaggregated area/gateway kind/bug
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 20h
Merged PRs (30d)
98

Description

### Describe the bug

In disaggregated mode the SGLang prefill is fired asynchronously and its result is never checked, so a failed prefill is invisible to the request path and the decode request is dispatched anyway.

In `doPrefillRequest` (`pkg/plugins/gateway/algorithms/pd_prefill_request.go`), the SGLang branch runs the prefill HTTP call in a goroutine and only logs on error:

```go
case SGLangEngine:
go func() {
defer r.prefillRequestTracker.RemovePrefillRequest(routingCtx.RequestID)
if _, err := r.executeHTTPRequest(apiURL, routingCtx, payload); err != nil {
klog.ErrorS(err, "prefill_request_failed", ...)
return // failure swallowed
}
...
}()
```

`Route()` does not wait for this goroutine — for SGLang `doPrefillRequest` returns `nil` immediately, then `Route()` sets the decode pod and returns. The decode request already carries the `bootstrap_room` generated in `preparePrefillPayload`. If the prefill 5xx'd, timed out, or never brought up its bootstrap server, the decode pod waits on a `bootstrap_room` rendezvous that never completes and only fails on SGLang's own bootstrap timeout. The client sees a long hang followed by an opaque decode-side error, with no gateway-side signal and no fast failure.

Two consequences:

- This is asymmetric with the vLLM path, which runs the prefill synchronously (`handleSyncPrefill`) and fails the request fast when the prefill errors.
- `Route()` emits `GatewayPrefillRequestSuccessTotal` for SGLang unconditionally, because the async call returns `nil` before the prefill has actually completed — so the success metric does not reflect prefill outcome.

### Steps to Reproduce

1. Disaggregated SGLang deployment (prefill + decode rolesets) with the `pd` router.
2. Make the selected prefill pod fail the prefill as a request arrives (e.g. kill/OOM it, or block its bootstrap port).
3. Send a request → it is not failed fast; the decode pod hangs until SGLang's bootstrap timeout and then returns an opaque error. The gateway logs `prefill_request_failed` but still counts prefill success and proceeds.

### Expected behavior

A failed SGLang prefill should fail (or retry) the request rather than dispatching a decode that will hang. Options:

- gate decode dispatch on the prefill bootstrap being acknowledged / the prefill request succeeding,
- retry on another prefill pod before giving up, and/or
- surface a fast, typed error to the client.

At minimum, prefill failure should be reflected in the request outcome and in metrics (`GatewayPrefillRequestSuccessTotal` should not be emitted for SGLang when the prefill has not been confirmed).

### Environment

AIBrix: main · SGLang prefill/decode (disaggregated, bootstrap mode).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.