nebari-dev / nebari-dev/llm-serving-pack

Implement envoyAIGateway.install flag in Helm chart

Open
#44 3 comments 0 reactions 1 assignee View on GitHub

@marcelovilla is already working on this.

Since Aug 27, 2026.

priority: medium ⚡ type: enhancement 💅🏼
Dominant language
Go
Stars
3
Forks
3
Avg merge
16h 31m
Merged PRs (30d)
11

Description

Problem

The chart's values.yaml has a placeholder for installing the Envoy AI Gateway:

envoyAIGateway:
  # install: if true, installs the Envoy AI Gateway (not yet implemented)
  install: false

But no templates actually do anything when this is set to true. The AI Gateway is required for the pack's core functionality (token counting, rate limiting, API key enforcement via SecurityPolicies, AIGatewayRoutes) and the operator generates AIGatewayRoute and SecurityPolicy resources that depend on the AI Gateway CRDs being installed on the cluster.

Today users must install the AI Gateway manually as separate ArgoCD apps or Helm releases.

Envoy AI Gateway details (upstream)

Based on https://aigateway.envoyproxy.io/docs/getting-started/installation/ and https://github.com/envoyproxy/ai-gateway:

  • OCI registry: oci://docker.io/envoyproxy
  • Two charts:
    • ai-gateway-crds-helm (CRDs: AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig, MCPRoute)
    • ai-gateway-helm (controller + webhooks)
  • Current stable version: v0.5.0 (as of January 2026)
  • Install namespace: envoy-ai-gateway-system
  • Prerequisites:
    • Kubernetes 1.32+
    • Envoy Gateway v1.6+ with the AI Gateway-compatible values applied (enables Extension Manager pointing at aieg.envoy-ai-gateway-system.svc.cluster.local:1063)

Proposed implementation

Add an optional sub-chart (or Helm template) that installs the AI Gateway when envoyAIGateway.install: true. Because there are two charts, the pack would need to install both.

Option A: Helm sub-chart dependencies

Add two dependencies in charts/nebari-llm-serving/Chart.yaml, both conditional on envoyAIGateway.install:

dependencies:
  - name: ai-gateway-crds-helm
    version: \"0.5.0\"
    repository: \"oci://docker.io/envoyproxy\"
    condition: envoyAIGateway.install
  - name: ai-gateway-helm
    version: \"0.5.0\"
    repository: \"oci://docker.io/envoyproxy\"
    condition: envoyAIGateway.install

Pros: reuses the upstream charts, gets updates for free.
Cons: the pack has less control over install ordering/timing. In ArgoCD, CRDs must install before the controller. Helm normally handles ordering but cross-chart dependencies in sub-charts can be finicky.

Option B: Ship two templated ArgoCD Applications

Under charts/nebari-llm-serving/templates/envoy-ai-gateway/, render two Application resources (one for CRDs, one for the controller) with sync-waves that enforce ordering. This is the pattern the pack already uses for other foundational resources.

Recommendation: Option B. The pack is fundamentally ArgoCD-driven and already ships ArgoCD Applications as templates. This gives explicit control over sync ordering (CRDs first) and fits the rest of the chart's conventions. Sub-chart dependencies would fight the ArgoCD lifecycle.

Values to expose

envoyAIGateway:
  install: false
  version: v0.5.0                          # pinned AI Gateway version (both charts use same tag)
  namespace: envoy-ai-gateway-system
  # Optional override of the OCI registry for air-gapped environments
  registry: oci://docker.io/envoyproxy

Envoy Gateway compatibility

The AI Gateway requires Envoy Gateway to be configured with the Extension Manager enabled, pointing at the AI Gateway controller service. Upstream provides a values file:

https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/manifests/envoy-gateway-values.yaml

nebari-infrastructure-core installs Envoy Gateway as a foundational service but may not use these values by default. We may need to either:

  1. Document that users must apply these values to their Envoy Gateway install, or
  2. File an issue against nebari-infrastructure-core to make the Envoy Gateway values configurable/aware of the AI Gateway extension, or
  3. Install our own Envoy Gateway on top of the nic one (not recommended)

Option 2 is cleanest long-term. Option 1 is fine for the initial implementation.

Acceptance criteria

  • Setting envoyAIGateway.install: true in Helm values installs both the CRDs chart and the controller chart, in the correct order (CRDs first).
  • The operator's existing AIGatewayRoute and SecurityPolicy generation works against the installed CRDs without further config.
  • When install: false, nothing extra is deployed and the chart installs normally (assumes the user has installed the AI Gateway separately or via another chart/ArgoCD app).
  • Documented in the README with a version compatibility note and the Envoy Gateway extension manager caveat.
  • Working example in examples/envoy-ai-gateway.yaml (shipped today as a workaround, should continue to work after the feature lands for users who prefer manual installation).

Workaround until implemented

See examples/envoy-ai-gateway.yaml for a two-Application ArgoCD manifest that installs both charts. Or via Helm directly:

# 1. CRDs first
helm upgrade -i aieg-crd oci://docker.io/envoyproxy/ai-gateway-crds-helm \\
  --version v0.5.0 \\
  --namespace envoy-ai-gateway-system \\
  --create-namespace

# 2. Envoy Gateway with AI-Gateway-compatible values (if not already done)
helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \\
  --version v1.6.0 \\
  --namespace envoy-gateway-system \\
  --create-namespace \\
  -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/main/manifests/envoy-gateway-values.yaml

# 3. Controller
helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \\
  --version v0.5.0 \\
  --namespace envoy-ai-gateway-system

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.