api7 / api7/aisix

P1-4: Routing strategies — latency-based + cost-based + tag-based

Open
#54 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

gap-with-litellm P1
Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

Problem

`crates/aisix-proxy/src/routing.rs` ships failover / round-robin / weighted (first-hop only). LiteLLM has 10+ strategies. Three of them cover the bulk of customer requests:

  1. latency-based — pick lowest-EWMA-latency target.
  2. cost-based — pick cheapest `usd_per_1k_token` target meeting a max-latency-tolerance.
  3. tag-based — match request header tags to target tags (`x-aisix-tag: prod` → only targets tagged prod).

Scope

DP (moonming/ai-gateway)
  • Extend `RoutingStrategy` enum in `aisix-core/src/models/routing.rs`.
  • Per-target rolling latency window (already partially tracked for routing.rs's existing fallback). Use `hdrhistogram` or simple EWMA.
  • Cost catalog in DP-side (or pull per-target cost from CP-pushed config).
  • Tag matching: request → `Vec` tags; target → `Vec` tags; intersection non-empty = eligible.
CP (api7/api7ee-3-control-plane)
  • `routing` resource (env-scoped) extends with new strategy enum values.
  • Per-target metadata: tags, cost (already partially there).
Dashboard UI (api7/AISIX-Cloud)
  • Routing-strategy picker adds 3 new entries.
  • Per-target form: tags input, cost field.
e2e
  • 3 targets with different EWMAs → assert latency-based picks the fastest.
  • 3 targets with different costs → cost-based picks cheapest.
  • Tag header → tag-based filters correctly.

Estimate

DP 3d, CP 1d, UI 1d, e2e 1d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/aisix-proxy/src/routing.rs and aisix-core/src/models/routing.rs to understand the existing strategies and latency tracking. Then inspect the CP routing resource and dashboard routing-strategy picker for their current target metadata. Done means the three strategies work across DP, CP, and UI, with e2e coverage for latency, cost, and tag selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.