[Testing] Test coverage gap analysis vs LiteLLM — DP supplementation roadmap

Context

Against the most mature LLM gateway in the wild — LiteLLM (BerriAI/litellm) — we audited the ai-gateway test suite end-to-end and identified concrete gaps. This is the meta-tracking issue for the supplementation roadmap.

Companion CP issue: api7/AISIX-Cloud#196 (cross-repo coordination).
Full plan document is kept in our internal knowledge base: test-plan-litellm-parity.md.

Snapshot

Strengths: 572 embedded units, wiremock isolation, Bedrock guardrail fail-open/fail-closed matrix, RPM/TPM/concurrency three-axis rate limit.
Weaknesses: thin E2E, narrow provider matrix, shallow streaming depth, missing error-recovery state machines.

🔴 P0 — production blockers

• G1 — Streaming + token accumulation accuracy (verified done on origin/main 56eaa31 — PR #394 (openai-tools-roundtrip + streaming-usage-accumulation e2e))

  • Site: crates/aisix-proxy/src/streaming_tests.rs
  • Assert: MockServer pushes N SSE chunks; final token count == non-streaming same input.
  • Ref: LiteLLM test_stream_chunk_builder.py

• G2 — Streaming + tool_calls delta assembly (verified done on origin/main 56eaa31 — PR #394 (openai-tools-roundtrip-e2e))

  • Site: same file as G1
  • Assert: tool_calls.arguments correctly concatenated across chunks
  • Ref: LiteLLM streaming function-call tests

• G3 — Circuit breaker / failure recovery

  • Site: routing/circuit_breaker_tests.rs
  • Assert: 5 consecutive 5xx → open → half-open → closed state machine
  • Ref: LiteLLM test_router.py::test_router_failures

• G4 — Embeddings endpoint full path

  • Site: crates/aisix-proxy/src/embeddings.rs (currently no unit tests)
  • Ref: LiteLLM test_embedding.py

• G5 — Cross-provider error envelope normalization

  • Assert: Anthropic 5xx → gateway emits OpenAI-shaped error.type / error.code
  • Ref: LiteLLM test_completion.py::test_normalised_error

• G6 — Weighted routing distribution validation

  • Assert: 100 requests with weight=70/30 land in [60,80] / [20,40]
  • Ref: LiteLLM test_router_weighted.py

• G7 — Expand E2E from 1 case to ≥10

  • Site: tests/e2e/src/cases/
  • Cover: streaming, guardrail, cache, ratelimit, budget, fallback
  • Ref: LiteLLM whole proxy_admin_ui_tests/

• G15 — OpenAI SDK reverse-call E2E (client compatibility)

  • Site: tests/e2e/src/cases/openai_sdk_compat/
  • Pattern: instantiate OpenAI(base_url="http://localhost:<gw_port>", api_key="sk-...") and AsyncOpenAI(...) clients; exercise the gateway through real OpenAI SDK paths (not handcrafted HTTP)
  • Cover: chat completions (sync + async), streaming, tool_calls, embeddings, vision messages
  • Why this matters: ai-gateway currently has only 1 Vitest smoke. Reverse-SDK calls catch wire-format mismatches that handwritten HTTP requests miss — header normalization, auto-retry behavior, streaming chunk parsing on the client side, async client lifecycle
  • Industry-practice basis: LiteLLM's proxy E2E (e.g. tests/test_openai_endpoints.py) overwhelmingly uses this pattern — AsyncOpenAI(base_url="http://0.0.0.0:4000") accounts for ~20% of all proxy E2E tests. It's the industry-standard way to prove "any OpenAI-compatible client just works"

🟡 P1 — core confidence

• G8 — Vision message full flow (verified done on origin/main 56eaa31 — vision-messages-e2e.test.ts) (image_url / base64 / detail param → upstream serialization)
• G9 — Rerank endpoint (verified done on origin/main 56eaa31 — rerank-e2e.test.ts) (Cohere rerank API → relevance score parsing)
• G10 — Semantic cache (embedding similarity) — currently only exact-match; need vector store backend tests
• G11 — Cache TTL / eviction edges (verified done on origin/main 56eaa31 — cache-ttl-eviction-e2e.test.ts) (TTL expiry, memory cap eviction, multi-instance consistency)
• G12 — Guardrail tag propagation (Bedrock returns piiDetected/toxicity → must surface in telemetry, not be swallowed)
• G13 — Sliding vs Fixed window rate-limit edges (race at window rollover)
• G14 — Cross-provider consistency parametrized matrix — rstest parametrized over provider × {chat, stream, tool, vision}

🟢 P2 — long tail

• Anthropic Messages native API pass-through (LiteLLM has a whole pass_through_endpoints/ directory)
• MCP protocol tests
• Realtime API (WebSocket path)
• CORS / size limits / header injection defense
• Files / Fine-tunes / Batches endpoints

🔗 CP↔DP coordination tests (need AISIX-Cloud counterpart)

• L1 Config change sub-second propagation: CP changes routing weight → DP picks up → real traffic ratio shifts — done: live weight-edit → traffic-shift pinned by weighted-routing-edit-e2e (PR #522); baseline propagation + initial weighted split already covered.
• L3 provider_key rotation with zero in-flight disruption — done: provider-key-rotation-e2e (PR #523) pins rotate-under-load liveness + revision bump. Audit confirmed zero-disruption is an architectural guarantee (per-request ArcSwap snapshot reads; no per-PK client to tear down); the 'rotated secret reaches upstream' facet is #220.
• [~] L4 DP restart state recovery — LOW priority: a DP cold-start pulling config from CP is already exercised by every startDP-based e2e (waitForDPReady proves it serves after boot); the distinct "restart an already-running DP" facet is marginal over that. Offline-degradation + recovery after a CP partition is covered by AISIX-Cloud offline_resilience (A5).
• L6 API key immediate revocation: CP revokes → DP rejects within 5s — done: apikey-revocation-live.spec.ts now bounds the propagation elapsed (PR AISIX-Cloud#695); DP auth is snapshot/watch-driven (not a TTL cache — the 5s constant is the budget cache), real propagation ~2ms, bound 20s for CI slack.
• L7 Guardrail decision-billing consistency: input-block doesn't bill, output-block bills already-sent tokens — done: pinned on both sides (DP guardrail-keyword/guardrail-output e2e; CP guardrail_billing_test.go). ACS billed_units → usage_events is the remaining product gap (AISIX-Cloud#659).

Roadmap

• Q1: Land G1 (streaming token accuracy) + G3 (circuit breaker) + G4 (embeddings) — closes the three biggest blind spots.
• Q2: Build a tests/integration/ level CP+DP harness (docker-compose + fixtures), exercise L1-L7.
• Q3: Adopt LiteLLM-style parametrized provider matrix; E2E case count target: 50+.

Key LiteLLM reference points

• tests/llm_translation_tests/ — provider format translation
• tests/router_unit_tests/ — routing strategies (least-busy, latency, weighted, simple-shuffle)
• tests/load_tests/ — perf/load (locust)
• tests/test_callbacks/ — 20+ observability integrations
• tests/guardrails_tests/ — chained guardrails

Status sweep — 2026-05-29 (verified against origin/main 56eaa31)

Code-verified the open boxes against current main (trackers had drifted):

• G1 / G2 / G8 / G9 / G11 — already DONE (ticked above): streaming-usage-accumulation-e2e, openai-tools-roundtrip-e2e (#394), vision-messages-e2e, rerank-e2e, cache-ttl-eviction-e2e.
• G3 (circuit breaker / half-open) — DESCOPED. Independent competitive review of LiteLLM + Portkey (primary source): neither ships a real open→half-open→closed FSM. LiteLLM uses an allowed-fails + cooldown-TTL deny-list (router_utils/cooldown_handlers.py); Portkey's breaker is hosted-only (OSS has hooks/constants only). AISIX already has the equivalent cooldown-TTL (crates/aisix-proxy/src/cooldown.rs), so this matches the industry norm. Not a real gap — recommend closing G3. Optional cheap future win: probabilistic re-admit near TTL expiry (not a full FSM).
• Still genuinely open (confirmed not-done, competitive review says peers DO them → worth building): G10 semantic cache (LiteLLM via redisvl/Qdrant), G12 guardrail-tag → telemetry (LiteLLM StandardLoggingGuardrailInformation), G13 rate-limit window (Portkey token-bucket; AISIX FixedWindowCounter has the rollover-burst race). Plus #226 audio/images usage (#406/#407) — LiteLLM prices audio by duration-seconds, images by (model,size,quality)×n.

Start by comparing the unchecked roadmap items with the current tree, especially routing/circuit_breaker_tests.rs, crates/aisix-proxy/src/embeddings.rs, and tests/e2e/src/cases/. Read the named LiteLLM references and existing Rust and Vitest tests before selecting one independently scoped gap. Done means the selected behavior has focused tests, passes the relevant test command, and the roadmap status is updated.