nebari-dev / nebari-dev/data-science-pack

Nebi token exchange fails with `invalid_client` when the chart is not deployed in a `jupyterhub` namespace

Open Beginner friendly
#248 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
7
Avg merge
1d 20h
Merged PRs (30d)
13

Description

Description

Ran into this while testing the pack on a local cluster, deployed in a namespace called data-science.

The hub's nebi token exchange (03-nebi-envs.py) authenticates to Keycloak with a client id derived as jupyterhub-<fullname> (the hubClientID helper): https://github.com/nebari-dev/data-science-pack/blob/f7e2305aebc05e65affbfbe4d3cee4bab4063d09/templates/_helpers.tpl#L227-L234. The values comment says this matches the operator naming convention, but the operator names clients <namespace>-<nebariapp-name> (https://github.com/nebari-dev/nebari-operator/blob/41258e255d5b771badbe122d3b17f4dfd5b1a6f5/internal/controller/utils/naming/naming.go#L74-L79). The two only match when the chart is deployed in a namespace called jupyterhub.

In any other namespace the hub presents a client id that does not exist in Keycloak, with the secret of a different client, and the exchange fails with invalid_client. User pods then get no NEBI_AUTH_TOKEN, the per-pod nebi runs unauthenticated in local mode, and everything it creates is owned by local-user instead of the real user.

The nebi client id has the same problem but it happens to work because nebi-<releaseName>-nebari-nebi-pack matches <namespace>-<fullname> given nebi-pack's namespace is nebi.

Expected behavior

The exchange works in any namespace. Suggested fix, two lines in _helpers.tpl:

  • hubClientID: derive with the actual namespace, printf "%s-%s" .Release.Namespace (include "nebari-data-science-pack.fullname" .).
  • nebiClientID: same problem waiting to fire, the prefix hardcodes nebi-pack's namespace. Derive as printf "%s-%s-nebari-nebi-pack" .Values.nebi.namespace .Values.nebi.releaseName (the chart already has nebi.namespace). Today both produce the same string because the namespace is nebi, which is why this half hasn't bitten yet.

Optionally, harden the hub's own id further by reading it from the operator's Secret: the chart already pulls client-secret from it into JUPYTERHUB_OIDC_CLIENT_SECRET, and the same Secret carries a client-id key. Reading both from the same place survives any future change to the operator's naming. That doesn't work for the nebi id (its Secret lives in the nebi namespace), so the helper fix is needed regardless.

How to reproduce

Deploy the chart on a NIC cluster in a namespace not called jupyterhub, log in, start a server, and check the hub logs. Deployments in a jupyterhub namespace don't reproduce it.

Command output

token-exchange step 1 FAILED: response={"error":"invalid_client","error_description":"Invalid client or Invalid client credentials"} (url=.../protocol/openid-connect/token, client_id=jupyterhub-data-science-pack-nebari-data-science-pack)
token-exchange: aborting — step 1 (refresh) returned no token
Nebi token exchange returned no JWT for marcelo

Verified against Keycloak's admin API: jupyterhub-data-science-pack-nebari-data-science-pack does not exist in the realm, data-science-data-science-pack-nebari-data-science-pack does. The mounted Secret's client-id key holds the latter, and those credentials authenticate fine when tested directly. Setting jupyterhub-client-id to the Secret's value fixed the exchange and workspace ownership.

Anything else?

Workaround until fixed:

jupyterhub:
  custom:
    jupyterhub-client-id: <namespace>-<release>-nebari-data-science-pack

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in templates/_helpers.tpl at the hubClientID and nebiClientID helpers, then inspect 03-nebi-envs.py and the referenced operator Secret values. Deploy the chart in a namespace other than jupyterhub, start a server, and check the hub logs; done means token exchange succeeds and the pod receives NEBI_AUTH_TOKEN with workspace ownership assigned to the real user.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, python
Domain
authentication, devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.