apache / apache/tooling-llmao

p6: backup for the LiteLLM database and fleet state

Open
#19 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

With fleet state in LiteLLM, there is one artifact rather than two.

```bash
sudo -u postgres pg_dump litellm | gzip > "$dest/litellm-$stamp.sql.gz"
```

**`$dest` is the open question.** Three constraints:

- the destination must be captured **off-host**. `bpc_client_asf` pulls via
rsync from a central BackupPC server, so the client has no local daemon and
coverage cannot be confirmed from the box. The share list lives on the
server — ask Infra which paths are pulled rather than assuming. A default
share of `/etc` and `/home` would miss `/var/backups`
- confirm the host is not in `bpc_client_asf::excludelist`
- a file-level copy of a live Postgres data directory is **not** a valid
backup; the `pg_dump` is what makes it restorable
What is at stake, and it grew with "LiteLLM routes have no api_base": Postgres now holds **routing** as well
as keys, budgets and spend. Losing it means the proxy no longer knows what to
proxy, not merely who may call it. Virtual keys are stored hashed and shown
once at mint, so recovery also means re-minting every key and reconfiguring
every consumer.

With fleet state in LiteLLM there is one artifact rather than two, and nothing
outside the database to reconstruct from. The GPU boxes keep serving from their
local assignment, so a lost database does not stop inference — but nothing can
re-fetch, re-route or re-authenticate until it is restored.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named. Start by asking Infra which paths bpc_client_asf pulls, then check that the host is not in bpc_client_asf::excludelist. Done means choosing an off-host destination for the pg_dump artifact and confirming the backup is restorable without copying the live PostgreSQL data directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, postgresql
Domain
databases, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.