nebari-dev / nebari-dev/llm-serving-pack

Helm chart install validation in CI (not just lint)

Open
#68 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: ci 👷🏽‍♀️ area: testing ✅ priority: medium ⚡ type: maintenance 🛠
Dominant language
Go
Stars
3
Forks
3
Avg merge
16h 31m
Merged PRs (30d)
11

Description

Problem

The current Lint Helm chart CI job runs helm lint and helm template, which only catches structural problems with the chart. It does not catch:

  • Templates that render valid YAML but fail to install on a real cluster (CRD ordering, RBAC issues, missing CRDs, namespace assumptions).
  • Operator startup crashes once the rendered manifests are actually applied.
  • Webhook bootstrap problems (cert injection, conversion, etc.).
  • Helm hook failures.

Issue #59 (collapse the pack into a single namespace) was caught at code review but the actual install would have failed in several ways the lint job missed. The chart having a separate namespace.yaml template was discovered only because a human read the diff.

Proposal

Add a CI job that:

  1. Spins up a kind (or k3d) cluster in the GitHub Actions runner.
  2. Installs the prerequisites the chart documents (cert-manager, gateway-api CRDs, Envoy Gateway + AI Gateway, KServe gateway-api-inference-extension CRDs).
  3. Runs helm install nebari-llm-serving ./charts/nebari-llm-serving --create-namespace -n nebari-llm-serving-system.
  4. Waits for operator and key-manager pods to become Ready.
  5. Applies a sample LLMModel CR (the one in examples/ or a CI-only minimal one).
  6. Asserts the operator reconciles it without error events and the expected SecurityPolicies / HTTPRoutes get created.
  7. Tears the cluster down.

This does not exercise actual model serving (no GPUs in the runner), but it catches every "the chart broke at install time" class of bug.

Notes

  • The job will be slower than lint - probably 5-8 minutes. Worth it.
  • Cache the prerequisite chart pulls between runs.
  • If kind setup is too expensive for every PR, gate this on paths filters touching charts/, operator/config/, or examples/.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing Lint Helm chart GitHub Actions job and the chart under charts/nebari-llm-serving; review the prerequisite versions and install commands documented by the project. Use the sample LLMModel in examples/ or a minimal CI fixture, and consider the paths operator/config/ and examples/ for CI gating. Done means a kind or k3d job installs the chart, waits for the operator and key-manager, verifies reconciliation plus SecurityPolicies and HTTPRoutes, and tears the cluster down.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, helm, kubernetes
Domain
ci-cd, devops, infrastructure, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.