vllm-project / vllm-project/aibrix
SGLang PD: prefill failure is silent and leaves the decode request hanging on bootstrap
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 98
Description
### Describe the bug
In disaggregated mode the SGLang prefill is fired asynchronously and its result is never checked, so a failed prefill is invisible to the request path and the decode request is dispatched anyway.
In `doPrefillRequest` (`pkg/plugins/gateway/algorithms/pd_prefill_request.go`), the SGLang branch runs the prefill HTTP call in a goroutine and only logs on error:
```go
case SGLangEngine:
go func() {
defer r.prefillRequestTracker.RemovePrefillRequest(routingCtx.RequestID)
if _, err := r.executeHTTPRequest(apiURL, routingCtx, payload); err != nil {
klog.ErrorS(err, "prefill_request_failed", ...)
return // failure swallowed
}
...
}()
```
`Route()` does not wait for this goroutine — for SGLang `doPrefillRequest` returns `nil` immediately, then `Route()` sets the decode pod and returns. The decode request already carries the `bootstrap_room` generated in `preparePrefillPayload`. If the prefill 5xx'd, timed out, or never brought up its bootstrap server, the decode pod waits on a `bootstrap_room` rendezvous that never completes and only fails on SGLang's own bootstrap timeout. The client sees a long hang followed by an opaque decode-side error, with no gateway-side signal and no fast failure.
Two consequences:
- This is asymmetric with the vLLM path, which runs the prefill synchronously (`handleSyncPrefill`) and fails the request fast when the prefill errors.
- `Route()` emits `GatewayPrefillRequestSuccessTotal` for SGLang unconditionally, because the async call returns `nil` before the prefill has actually completed — so the success metric does not reflect prefill outcome.
### Steps to Reproduce
1. Disaggregated SGLang deployment (prefill + decode rolesets) with the `pd` router.
2. Make the selected prefill pod fail the prefill as a request arrives (e.g. kill/OOM it, or block its bootstrap port).
3. Send a request → it is not failed fast; the decode pod hangs until SGLang's bootstrap timeout and then returns an opaque error. The gateway logs `prefill_request_failed` but still counts prefill success and proceeds.
### Expected behavior
A failed SGLang prefill should fail (or retry) the request rather than dispatching a decode that will hang. Options:
- gate decode dispatch on the prefill bootstrap being acknowledged / the prefill request succeeding,
- retry on another prefill pod before giving up, and/or
- surface a fast, typed error to the client.
At minimum, prefill failure should be reflected in the request outcome and in metrics (`GatewayPrefillRequestSuccessTotal` should not be emitted for SGLang when the prefill has not been confirmed).
### Environment
AIBrix: main · SGLang prefill/decode (disaggregated, bootstrap mode).
Contributor guide
Assessment
This issue has not been assessed yet.