apache / apache/tooling-llmao

Fleet management in the UI

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Three actions on `/fleet`:

- **Add host** — IP, label, provider, instance id, `[model, port]` rows.
Validate the model exists in the catalog and the port is unclaimed on that
IP; generate the vLLM api_key; record as a pending assignment. No route yet
— see "Health-gated route registration".
- **Retire host** — delete the route and drop the assignment. A box still
running then gets 404 from `/vllm/config` on its next fetch, which is the
correct signal that it is no longer ours.
- **Edit servers** — change the model/port rows for a box being repurposed.
Note this is boot-lifetime config: it takes effect when the box restarts
vLLM, not immediately. Surface that rather than implying it is live.
Label, provider, instance id and timestamps go in `model_info` on the route.

Retirement currently loses its record, since deleting a route deletes the row.
Either mark `model_info.retired` and remove from routing, or accept LiteLLM's
audit trail. Worth deciding rather than losing by default.

`last_config_fetch` should render as a coloured age — green under an hour, red
over a week. **Never auto-retire:** a box down for maintenance is
indistinguishable from one that is dead.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the `/fleet` UI and the existing `/vllm/config` route, then trace how assignments and `model_info` are stored. Decide how retirement is represented before implementing Add host, Retire host, and Edit servers, including health-gated routing and the boot-lifetime warning. Done means the three actions work, validation and pending assignments are handled, retirement does not silently lose history, and `last_config_fetch` renders the specified age colors without auto-retiring hosts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.