vllm-project / vllm-project/aibrix

[TEST] Add Gateway Plugin integration and functional regression coverage

Open
#2,708 2 comments 0 reactions 1 assignee Claimed by @googs1025 View on GitHub
area/gateway area/kv-cache area/testing help wanted kind/bug
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

## Status

The initial Gateway integration foundation and representative regression coverage were merged in [PR #2716](https://github.com/vllm-project/aibrix/pull/2716).

The merged PR added 23 Gateway integration specs. The cases below are **not included yet** and are available for follow-up contributors.

## Already covered by #2716

- Real ext_proc `Server.Process` integration fixture.
- Isolated fake cache, metrics observer, stream, RouterManager, and prefix indexer.
- Representative routing cases for `random`, `least-request`, `least-kv-cache`, `least-latency`, `load-balance`, and `prefix-cache`.
- Readiness filtering, `external-filter`, invalid strategy, zero-weight, and one multi-strategy case.
- Config profile/request-header precedence.
- Request body/header transformation and OpenAI-compatible error envelope.
- Prefix-cache miss-to-hit with fixture-local state.
- SSE callback boundary, deadline/cancellation cleanup, in-flight metrics, and exactly-once finalization.

## Help wanted: remaining cases

Please open a focused follow-up PR and check off the case(s) it implements.

### Gateway integration follow-ups

These should run through the existing `test/integration/gateway` fixture without Envoy, Kind, GPUs, or real model inference.

#### Routing and Pod state

- [ ] Missing metrics and fallback behavior for each relevant strategy family.
- [ ] Full weighted multi-strategy causality matrix:
verify that changing weights changes the selected Pod when individual strategies prefer different Pods.
- [ ] Prefix-cache plus load-balance interaction.
- [ ] Load-imbalance protection with a dedicated imbalance fixture.
- [ ] Pod removal/replacement during an in-flight routing decision.
- [ ] Additional readiness transitions beyond the current static not-ready filter.
- [ ] Multi-port/data-parallel routing cases that can be exercised without a cluster.

#### Request and response contracts

- [ ] Request header preservation/filtering matrix, including headers that must survive to Envoy.
- [ ] Request body preservation across multiple processing callbacks.
- [ ] Additional upstream OpenAI-compatible error shapes and status/body combinations.
- [ ] Model availability failure does not consume rate-limit state incorrectly.
- [ ] Request accounting remains exactly once across every early-return error path.

#### Streaming

- [ ] Upstream stream error after one or more successful chunks.
- [ ] Client cancellation after streaming has started.
- [ ] Upstream EOF before `[DONE]`.
- [ ] Multiple SSE events in one callback chunk.
- [ ] UTF-8 content split across callback chunks.
- [ ] Empty end-of-stream chunks.
- [ ] Usage extraction from a final streaming chunk while preserving the original body.
- [ ] Streaming metrics and finalization for each error/EOF path.

#### Resilience

- [ ] Context cancellation and deadline coverage for additional callback positions.
- [ ] Shutdown while `Recv` is blocked.
- [ ] Repeated terminal/error callbacks do not double-finalize.

### Gateway E2E follow-ups

These require Envoy/Kind and should reuse the shared framework and recorder from [#2675](https://github.com/vllm-project/aibrix/issues/2675).

- [ ] Verify the actual backend HTTP request body and headers after Gateway transformation.
- [ ] Compare the Gateway target-Pod headers with the Pod that actually receives the request.
- [ ] Retry behavior with a failing first backend and successful retry.
- [ ] Backend timeout, connection failure, and error propagation.
- [ ] Readiness changes observed through the deployed Gateway.
- [ ] External Pod filtering through the deployed Gateway.
- [ ] Pod removal/replacement during routing.
- [ ] Streaming success, upstream error, EOF, and cancellation through Envoy.
- [ ] Multi-Gateway replica state synchronization.
- [ ] Config profile precedence in the deployed Gateway.
- [ ] Multi-port/data-parallel routing in a deployed environment.

## How to contribute

- Pick one small, coherent group of unchecked cases.
- Add a named test or contract; do not silently change an existing expectation.
- Reuse the existing Gateway integration fixture and keep test state isolated.
- Use the #2675 E2E recorder when the assertion requires the actual backend request.
- Include request ID, model, strategy, target Pod, callback sequence, and diagnostics in failures.
- Assert functional behavior only; do not add fixed TTFT/TPOT, latency, or throughput thresholds.
- Update this checklist in the follow-up PR description or issue comment.

## Related work

- #2675 owns E2E restructuring, shared framework/recorder, and PD HTTP handoff contracts.
- #2716 contains the merged Gateway integration foundation and initial regression coverage.
- PD protocol contract tests should not be duplicated here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.