P1-4: Routing strategies — latency-based + cost-based + tag-based
Open
Nobody has claimed this yet.
gap-with-litellm
P1
- Dominant language
- Rust
- Stars
- 157
- Forks
- 32
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 145
Description
Problem
`crates/aisix-proxy/src/routing.rs` ships failover / round-robin / weighted (first-hop only). LiteLLM has 10+ strategies. Three of them cover the bulk of customer requests:
- latency-based — pick lowest-EWMA-latency target.
- cost-based — pick cheapest `usd_per_1k_token` target meeting a max-latency-tolerance.
- tag-based — match request header tags to target tags (`x-aisix-tag: prod` → only targets tagged prod).
Scope
DP (moonming/ai-gateway)
- Extend `RoutingStrategy` enum in `aisix-core/src/models/routing.rs`.
- Per-target rolling latency window (already partially tracked for routing.rs's existing fallback). Use `hdrhistogram` or simple EWMA.
- Cost catalog in DP-side (or pull per-target cost from CP-pushed config).
- Tag matching: request → `Vec` tags; target → `Vec` tags; intersection non-empty = eligible.
CP (api7/api7ee-3-control-plane)
- `routing` resource (env-scoped) extends with new strategy enum values.
- Per-target metadata: tags, cost (already partially there).
Dashboard UI (api7/AISIX-Cloud)
- Routing-strategy picker adds 3 new entries.
- Per-target form: tags input, cost field.
e2e
- 3 targets with different EWMAs → assert latency-based picks the fastest.
- 3 targets with different costs → cost-based picks cheapest.
- Tag header → tag-based filters correctly.
Estimate
DP 3d, CP 1d, UI 1d, e2e 1d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crates/aisix-proxy/src/routing.rs and aisix-core/src/models/routing.rs to understand the existing strategies and latency tracking. Then inspect the CP routing resource and dashboard routing-strategy picker for their current target metadata. Done means the three strategies work across DP, CP, and UI, with e2e coverage for latency, cost, and tag selection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100