nebari-dev / nebari-dev/llm-serving-pack

Initial implementation of nebari-llm-serving-pack

Open
#1 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

epic
Dominant language
Go
Stars
3
Forks
3
Avg merge
16h 31m
Merged PRs (30d)
11

Description

Tracking issue for the initial implementation of the Nebari LLM serving pack.

See docs/design.md for the full design document.

Summary

This pack deploys a Go operator that watches LLMModel CRDs to manage LLM serving via llm-d. Each model gets:

  • vLLM serving pods with init container for model download
  • InferencePool + EPP for intelligent inference scheduling
  • External endpoint with API key auth
  • Internal endpoint with OIDC/JWT auth
  • Token counting and rate limiting via Envoy AI Gateway

An optional key manager web UI lets users generate API keys for models they have access to, based on their OIDC groups.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/design.md, the only file named in the issue, to understand the proposed pack architecture and implementation scope. Then identify the entry points for the Go operator, LLMModel CRDs, serving endpoints, authentication, and the optional key manager. Done means the initial pack provides the listed serving, scheduling, access-control, rate-limiting, and key-management capabilities.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend, infrastructure, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.