nebari-dev / nebari-dev/llm-serving-pack
Initial implementation of nebari-llm-serving-pack
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3
- Forks
- 3
- Avg merge
- 16h 31m
- Merged PRs (30d)
- 11
Description
Tracking issue for the initial implementation of the Nebari LLM serving pack.
See docs/design.md for the full design document.
Summary
This pack deploys a Go operator that watches LLMModel CRDs to manage LLM serving via llm-d. Each model gets:
- vLLM serving pods with init container for model download
- InferencePool + EPP for intelligent inference scheduling
- External endpoint with API key auth
- Internal endpoint with OIDC/JWT auth
- Token counting and rate limiting via Envoy AI Gateway
An optional key manager web UI lets users generate API keys for models they have access to, based on their OIDC groups.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/design.md, the only file named in the issue, to understand the proposed pack architecture and implementation scope. Then identify the entry points for the Go operator, LLMModel CRDs, serving endpoints, authentication, and the optional key manager. Done means the initial pack provides the listed serving, scheduling, access-control, rate-limiting, and key-management capabilities.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend, infrastructure, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100