Migrate integration test to action-nebari-sandbox (fixes Docker Hub timeout)

Summary

Migrate test-integration.yaml from its bespoke, hand-rolled platform bootstrap to the shared nebari-dev/action-nebari-sandbox action (platform profile) + the add-software-pack sub-action — the same pattern already used by nebari-nebi-pack and nebari-data-science-pack.

This repo is currently the outlier: it manually installs MetalLB, Envoy Gateway, cert-manager, Keycloak (via nebari-operator/dev/scripts/services/install.sh) and builds the nebari-operator from source, where the sibling packs get all of that from the shared action.

Motivation

The integration test is broken repo-wide and has been since ~2026-06-12:

Date	Branch	Duration	Result
May 14 – Jun 11	main & PR branches	~7–8 min	✅
Jun 12 onward	every branch (incl. unrelated PRs)	30m	❌ timeout

Root cause: the Install foundational services step (install.sh) helm installs Envoy Gateway, whose OCI chart pull from Docker Hub now blocks ~20 minutes (Docker Hub anonymous-pull throttling on shared GitHub-runner IPs), consuming the entire 30-min job budget before cert-manager or the chart ever install. Re-running does not help — it's deterministic, not flaky.

The shared action sidesteps this entirely: it provisions the foundational stack via NIC (median wall-clock is tracked in the action's README and is in the single-digit minutes), and pulls the operator image from quay.io, not Docker Hub.

Research / findings

The platform profile ships everything we currently hand-build. Confirmed in nebari-infrastructure-core:

• pkg/argocd/templates/apps/ includes cert-manager, envoy-gateway, keycloak, metallb, and nebari-operator.
• nebari-operator deploys from github.com/nebari-dev/nebari-operator.git/config/default?ref=v0.1.0-alpha.20, image quay.io/nebari/nebari-operator:0.1.0-alpha.20, into namespace nebari-operator-system — exactly the namespace/deployment our current kubectl wait targets. So the NebariApp CRD + controller are present out of the box.

What the migration deletes (≈90 lines of bootstrap):

• Create kind cluster → replaced by k3d via the action
• Install MetalLB + Configure IP pool → deleted; k3d's klipper servicelb provides the gateway-ip output
• Install foundational services (the 20-min hang) → provided by NIC
• Setup Keycloak realm → provided by NIC
• Build & load nebari-operator from source + deploy + wait → platform ships it

What carries over unchanged: all the verification steps (NebariApp Ready, HTTPRoutes ≥ 2, SecurityPolicies ≥ 2, dashboard port-forward health check) and the Debug-on-failure block.

Prerequisite: deterministic KubeRay CRD ordering (chart change)

The chart defaults kuberay-operator.enabled: true, so the KubeRay subchart's CRDs (crds/) and the RayService CR install in the same ArgoCD Application. Today's CI sidesteps ordering by helm install-ing KubeRay separately first; the GitOps path can't do that as cleanly.

Add a sync-wave to the RayService so it applies one wave after the CRDs + operator (wave 0). ArgoCD waits for a wave to be Healthy before the next, so the rayservices.ray.io CRD is Established and the controller is running before the CR is applied — deterministic, instead of leaning on SkipDryRunOnMissingResource + sync retries to absorb a first-sync race.

# chart/templates/rayservice.yaml
   annotations:
     argocd.argoproj.io/compare-options: IgnoreExtraneous
+    argocd.argoproj.io/sync-wave: "1"

Prototyped and verified: RayService renders with sync-wave: "1", the 3 KubeRay CRDs render at wave 0 (via --include-crds), helm lint passes. This is a standalone improvement that also hardens real production GitOps deploys.

Implementation sketch

1. chart/templates/rayservice.yaml — add the sync-wave annotation (prerequisite above).
2. New .github/ci/rayserve-application.yaml — ArgoCD Application modeled on nebi-pack-application.yaml: repoURL: "file://${GITOPS_DIR}", path: rayserve, helm values: with nebariapp.enabled=true, hostnames, and downsized resources.
3. Rewrite test-integration.yaml:
   • nebari-dev/action-nebari-sandbox@<sha> (pin to a v2 commit) with profile: platform
   • surface KUBECONFIG from the kubeconfig output
   • add-software-pack with app-name: rayserve, chart-source: ./chart (or an rsync-staged copy excluding .git, as nebi-pack does), application-manifest: ./.github/ci/rayserve-application.yaml
   • keep the existing readiness + verification + debug steps
   • cleanup: k3d cluster delete ${{ steps.sandbox.outputs.cluster-name }}

Decisions / risks to resolve during implementation

• Version-under-test shift: today we test nebari-operator HEAD; the platform pins v0.1.0-alpha.20. Arguably more correct (validate against the released platform), but a real change. If HEAD coverage is needed, override via nic-config / a custom app.
• helm template vs ArgoCD CRDs: bare helm template does not emit crds/ (needs --include-crds). ArgoCD passes --include-crds, so CRDs are managed there — but document this so future debugging doesn't conclude they're "missing."
• Application Healthy ≠ pods Ready: ArgoCD has no health check for ray.io/RayService, so it reports Healthy on sync, before Ray pods are up. The add-software-pack wait is necessary but not sufficient — keep the explicit kubectl wait pod -l ray.io/node-type=head --for=condition=ready gate.
• Auth-upgrade test: today it's a helm upgrade --set nebariapp.auth.enabled=true. Under GitOps decide between editing the Application's helm values + re-sync, or deploying auth-on from the start.
• Pin the action by commit SHA (siblings do), targeting v2.

Acceptance criteria

• test-integration.yaml uses action-nebari-sandbox (platform) + add-software-pack; no MetalLB/install.sh/operator-build steps remain.
• RayService sync-wave annotation landed; CRDs install before the CR deterministically.
• NebariApp / HTTPRoute / SecurityPolicy / dashboard verifications retained and green.
• Explicit Ray pod-readiness wait retained.
• Wall-clock back in the single-digit-minutes range; no Docker Hub timeout.

Start with chart/templates/rayservice.yaml, test-integration.yaml, and the sibling nebi-pack-application.yaml pattern. Review the existing integration workflow and confirm the action, application manifest, KubeRay sync ordering, readiness checks, and debug steps. Done means the acceptance criteria pass, including green verification steps and no Docker Hub timeout.