nebari-dev / nebari-dev/data-science-pack

Hub crash-loops on missing /etc/oauth secret instead of falling back to dummy auth

Open
#133 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

size: S
Dominant language
Python
Stars
5
Forks
7
Avg merge
1d 20h
Merged PRs (30d)
13

Description

What happened

When the operator-provisioned OIDC client Secret doesn't exist yet, the hub crash-loops at startup:

File "00-gateway-auth.py", line 631, in <module>
File "00-gateway-auth.py", line 565, in _read_secret_file
FileNotFoundError: [Errno 2] No such file or directory: '/etc/oauth/issuer-url'

The chart mounts the secret with optional: true, and the values.yaml comments read as though the chart degrades gracefully when Keycloak isn't wired ("Empty is OK if...", dummy authenticator as the default). In practice 00-gateway-auth.py reads /etc/oauth/issuer-url unconditionally; with the secret missing, kubelet mounts an empty directory and the read throws.

How we hit it

The NebariApp wasn't being reconciled because the namespace was missing the nebari.dev/managed=true label, so the operator never created the Secret. The operator's status condition (NamespaceNotOptedIn) pointed at the real cause, but the hub traceback is what a deployer sees first, and it doesn't hint at the Secret name or the NebariApp at all.

Once the label was fixed and the operator created the Secret, the crash-looping pod recovered on its own (kubelet re-syncs optional secret volumes), which is nice, but the intermediate state is confusing.

Suggestion

Either of these would have saved a debugging round-trip:

  1. Fall back to dummy auth with a loud warning when /etc/oauth is empty, matching what the comments imply, or
  2. Fail with an explicit message: "OIDC client secret not found; the nebari-operator has not provisioned it yet, check kubectl get nebariapp -n <ns> status".

Given the secret appearing later heals the pod automatically, option 2 with a clear message (still crashing, so the pod retries) may actually be the better behavior; option 1 risks briefly exposing a dummy-auth hub on a real deployment.

Seen on 0.1.0-alpha.16 with nebari-operator 0.1.0-alpha.20.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with 00-gateway-auth.py, especially _read_secret_file and the startup call shown in the traceback, then compare its behavior with the values.yaml comments and optional /etc/oauth mount. Decide whether missing files should produce a clear operator-oriented error or dummy auth, and verify the chosen behavior when the Secret is absent and after it appears.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
authentication, backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.