nebari-dev / nebari-dev/data-science-pack
Hub crash-loops on missing /etc/oauth secret instead of falling back to dummy auth
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5
- Forks
- 7
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 13
Description
What happened
When the operator-provisioned OIDC client Secret doesn't exist yet, the hub crash-loops at startup:
File "00-gateway-auth.py", line 631, in <module>
File "00-gateway-auth.py", line 565, in _read_secret_file
FileNotFoundError: [Errno 2] No such file or directory: '/etc/oauth/issuer-url'
The chart mounts the secret with optional: true, and the values.yaml comments read as though the chart degrades gracefully when Keycloak isn't wired ("Empty is OK if...", dummy authenticator as the default). In practice 00-gateway-auth.py reads /etc/oauth/issuer-url unconditionally; with the secret missing, kubelet mounts an empty directory and the read throws.
How we hit it
The NebariApp wasn't being reconciled because the namespace was missing the nebari.dev/managed=true label, so the operator never created the Secret. The operator's status condition (NamespaceNotOptedIn) pointed at the real cause, but the hub traceback is what a deployer sees first, and it doesn't hint at the Secret name or the NebariApp at all.
Once the label was fixed and the operator created the Secret, the crash-looping pod recovered on its own (kubelet re-syncs optional secret volumes), which is nice, but the intermediate state is confusing.
Suggestion
Either of these would have saved a debugging round-trip:
- Fall back to dummy auth with a loud warning when
/etc/oauthis empty, matching what the comments imply, or - Fail with an explicit message: "OIDC client secret not found; the nebari-operator has not provisioned it yet, check
kubectl get nebariapp -n <ns>status".
Given the secret appearing later heals the pod automatically, option 2 with a clear message (still crashing, so the pod retries) may actually be the better behavior; option 1 risks briefly exposing a dummy-auth hub on a real deployment.
Seen on 0.1.0-alpha.16 with nebari-operator 0.1.0-alpha.20.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with 00-gateway-auth.py, especially _read_secret_file and the startup call shown in the traceback, then compare its behavior with the values.yaml comments and optional /etc/oauth mount. Decide whether missing files should produce a clear operator-oriented error or dummy auth, and verify the chosen behavior when the Secret is absent and after it appears.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- authentication, backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100