nebari-dev / nebari-dev/data-science-pack
Nebi token exchange fails with `invalid_client` when the chart is not deployed in a `jupyterhub` namespace
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5
- Forks
- 7
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 13
Description
Description
Ran into this while testing the pack on a local cluster, deployed in a namespace called data-science.
The hub's nebi token exchange (03-nebi-envs.py) authenticates to Keycloak with a client id derived as jupyterhub-<fullname> (the hubClientID helper): https://github.com/nebari-dev/data-science-pack/blob/f7e2305aebc05e65affbfbe4d3cee4bab4063d09/templates/_helpers.tpl#L227-L234. The values comment says this matches the operator naming convention, but the operator names clients <namespace>-<nebariapp-name> (https://github.com/nebari-dev/nebari-operator/blob/41258e255d5b771badbe122d3b17f4dfd5b1a6f5/internal/controller/utils/naming/naming.go#L74-L79). The two only match when the chart is deployed in a namespace called jupyterhub.
In any other namespace the hub presents a client id that does not exist in Keycloak, with the secret of a different client, and the exchange fails with invalid_client. User pods then get no NEBI_AUTH_TOKEN, the per-pod nebi runs unauthenticated in local mode, and everything it creates is owned by local-user instead of the real user.
The nebi client id has the same problem but it happens to work because nebi-<releaseName>-nebari-nebi-pack matches <namespace>-<fullname> given nebi-pack's namespace is nebi.
Expected behavior
The exchange works in any namespace. Suggested fix, two lines in _helpers.tpl:
hubClientID: derive with the actual namespace,printf "%s-%s" .Release.Namespace (include "nebari-data-science-pack.fullname" .).nebiClientID: same problem waiting to fire, the prefix hardcodes nebi-pack's namespace. Derive asprintf "%s-%s-nebari-nebi-pack" .Values.nebi.namespace .Values.nebi.releaseName(the chart already hasnebi.namespace). Today both produce the same string because the namespace isnebi, which is why this half hasn't bitten yet.
Optionally, harden the hub's own id further by reading it from the operator's Secret: the chart already pulls client-secret from it into JUPYTERHUB_OIDC_CLIENT_SECRET, and the same Secret carries a client-id key. Reading both from the same place survives any future change to the operator's naming. That doesn't work for the nebi id (its Secret lives in the nebi namespace), so the helper fix is needed regardless.
How to reproduce
Deploy the chart on a NIC cluster in a namespace not called jupyterhub, log in, start a server, and check the hub logs. Deployments in a jupyterhub namespace don't reproduce it.
Command output
token-exchange step 1 FAILED: response={"error":"invalid_client","error_description":"Invalid client or Invalid client credentials"} (url=.../protocol/openid-connect/token, client_id=jupyterhub-data-science-pack-nebari-data-science-pack)
token-exchange: aborting — step 1 (refresh) returned no token
Nebi token exchange returned no JWT for marcelo
Verified against Keycloak's admin API: jupyterhub-data-science-pack-nebari-data-science-pack does not exist in the realm, data-science-data-science-pack-nebari-data-science-pack does. The mounted Secret's client-id key holds the latter, and those credentials authenticate fine when tested directly. Setting jupyterhub-client-id to the Secret's value fixed the exchange and workspace ownership.
Anything else?
Workaround until fixed:
jupyterhub:
custom:
jupyterhub-client-id: <namespace>-<release>-nebari-data-science-pack
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in templates/_helpers.tpl at the hubClientID and nebiClientID helpers, then inspect 03-nebi-envs.py and the referenced operator Secret values. Deploy the chart in a namespace other than jupyterhub, start a server, and check the hub logs; done means token exchange succeeds and the pod receives NEBI_AUTH_TOKEN with workspace ownership assigned to the real user.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, python
- Domain
- authentication, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 85/100