apache / apache/flink-agents

[Umbrella] Pluggable model routing (MODEL_ROUTER): delivery status, Python parity, documentation, and follow-ups

Open
#1,062 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
452
Forks
167
Avg merge
5d 9h
Merged PRs (30d)
49

Description

Umbrella issue to track pluggable in-chat model routing end to end, as requested in #1042.

**Design:** discussion #897 — select-not-delegate, the framework owns the model call; judge-style policies run on the engine's durable, metered chat path.

**Delivered in v1 (#964, merged):**
- `RoutingStrategy` SPI and the `MODEL_ROUTER` resource; `Strategies.rules(...)` and `Strategies.of(...)`
- Durable decision persistence and replay (ids `route:` / `chat::`), route-once-per-reasoning-loop
- Fallback through declared candidate order; `ModelRoutingEvent` (observability-only) with decision metadata and latency histogram
- Build-time validation (regex compile, null/template checks)

**v2 (#1042, in review):**
- Framework-managed LLM-as-judge: `Strategies.llm(judgeModel[, promptTemplate])`; the engine executes the judge call durably (`judge:`) with token attribution and events
- Verdicts constrained to candidate names; unparseable/non-candidate replies abstain to the default model; judge failures honor the error-handling strategy; `decision_source` recorded
- Plan-time validation of judge references; `ModelRoutingJudgeExample`

**Follow-ups to track:**
- [ ] User documentation for model routing (v1 + v2) — covers the `doc-needed` label
- [ ] Python-side parity (today `add_resource(..., MODEL_ROUTER)` raises with a clear error; cross-language plan deserialization is already tested)

Contributor guide

Open the contributing guide

Research direction

Start with design discussion #897, delivered work in #964, and the v2 review in #1042. Determine the documentation entry points for v1 and v2 and the Python path where add_resource(..., MODEL_ROUTER) currently raises; review the existing cross-language plan deserialization test. Done means user documentation exists and Python-side model-router parity is implemented and tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.