vllm-project / vllm-project/aibrix
[TEST] Add Gateway Plugin integration and functional regression coverage
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
## Status
The initial Gateway integration foundation and representative regression coverage were merged in [PR #2716](https://github.com/vllm-project/aibrix/pull/2716).
The merged PR added 23 Gateway integration specs. The cases below are **not included yet** and are available for follow-up contributors.
## Already covered by #2716
- Real ext_proc `Server.Process` integration fixture.
- Isolated fake cache, metrics observer, stream, RouterManager, and prefix indexer.
- Representative routing cases for `random`, `least-request`, `least-kv-cache`, `least-latency`, `load-balance`, and `prefix-cache`.
- Readiness filtering, `external-filter`, invalid strategy, zero-weight, and one multi-strategy case.
- Config profile/request-header precedence.
- Request body/header transformation and OpenAI-compatible error envelope.
- Prefix-cache miss-to-hit with fixture-local state.
- SSE callback boundary, deadline/cancellation cleanup, in-flight metrics, and exactly-once finalization.
## Help wanted: remaining cases
Please open a focused follow-up PR and check off the case(s) it implements.
### Gateway integration follow-ups
These should run through the existing `test/integration/gateway` fixture without Envoy, Kind, GPUs, or real model inference.
#### Routing and Pod state
- [ ] Missing metrics and fallback behavior for each relevant strategy family.
- [ ] Full weighted multi-strategy causality matrix:
verify that changing weights changes the selected Pod when individual strategies prefer different Pods.
- [ ] Prefix-cache plus load-balance interaction.
- [ ] Load-imbalance protection with a dedicated imbalance fixture.
- [ ] Pod removal/replacement during an in-flight routing decision.
- [ ] Additional readiness transitions beyond the current static not-ready filter.
- [ ] Multi-port/data-parallel routing cases that can be exercised without a cluster.
#### Request and response contracts
- [ ] Request header preservation/filtering matrix, including headers that must survive to Envoy.
- [ ] Request body preservation across multiple processing callbacks.
- [ ] Additional upstream OpenAI-compatible error shapes and status/body combinations.
- [ ] Model availability failure does not consume rate-limit state incorrectly.
- [ ] Request accounting remains exactly once across every early-return error path.
#### Streaming
- [ ] Upstream stream error after one or more successful chunks.
- [ ] Client cancellation after streaming has started.
- [ ] Upstream EOF before `[DONE]`.
- [ ] Multiple SSE events in one callback chunk.
- [ ] UTF-8 content split across callback chunks.
- [ ] Empty end-of-stream chunks.
- [ ] Usage extraction from a final streaming chunk while preserving the original body.
- [ ] Streaming metrics and finalization for each error/EOF path.
#### Resilience
- [ ] Context cancellation and deadline coverage for additional callback positions.
- [ ] Shutdown while `Recv` is blocked.
- [ ] Repeated terminal/error callbacks do not double-finalize.
### Gateway E2E follow-ups
These require Envoy/Kind and should reuse the shared framework and recorder from [#2675](https://github.com/vllm-project/aibrix/issues/2675).
- [ ] Verify the actual backend HTTP request body and headers after Gateway transformation.
- [ ] Compare the Gateway target-Pod headers with the Pod that actually receives the request.
- [ ] Retry behavior with a failing first backend and successful retry.
- [ ] Backend timeout, connection failure, and error propagation.
- [ ] Readiness changes observed through the deployed Gateway.
- [ ] External Pod filtering through the deployed Gateway.
- [ ] Pod removal/replacement during routing.
- [ ] Streaming success, upstream error, EOF, and cancellation through Envoy.
- [ ] Multi-Gateway replica state synchronization.
- [ ] Config profile precedence in the deployed Gateway.
- [ ] Multi-port/data-parallel routing in a deployed environment.
## How to contribute
- Pick one small, coherent group of unchecked cases.
- Add a named test or contract; do not silently change an existing expectation.
- Reuse the existing Gateway integration fixture and keep test state isolated.
- Use the #2675 E2E recorder when the assertion requires the actual backend request.
- Include request ID, model, strategy, target Pod, callback sequence, and diagnostics in failures.
- Assert functional behavior only; do not add fixed TTFT/TPOT, latency, or throughput thresholds.
- Update this checklist in the follow-up PR description or issue comment.
## Related work
- #2675 owns E2E restructuring, shared framework/recorder, and PD HTTP handoff contracts.
- #2716 contains the merged Gateway integration foundation and initial regression coverage.
- PD protocol contract tests should not be duplicated here.
Contributor guide
Assessment
This issue has not been assessed yet.