nebari-dev / nebari-dev/data-science-pack

jhub-apps service does not receive CA trust env vars: nebi environment selector silently empty behind a private CA

Open Beginner friendly
#187 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
7
Avg merge
1d 20h
Merged PRs (30d)
13

Description

Symptom

On a private-CA deployment, the "Software Environment" dropdown never appears in the jhub-apps create-app form. No error surfaces in the UI; the japps log shows:

token-exchange step 1 FAILED: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate ...> (url=https://keycloak.<domain>/realms/nebari/protocol/openid-connect/token, ...)
nebi-envs: token exchange returned no JWT for <user>
Found conda environments: []

Root cause

get_nebi_environments (03-nebi-envs.py) runs inside the japps managed service, and JupyterHub managed services spawn with a minimal environment — they do not inherit the hub container's env. 02-jhub-apps.py already forwards JUPYTERHUB_OIDC_CLIENT_SECRET into the japps service environment for exactly this reason, but the CA trust env vars are not forwarded. Verified on-cluster: hub pid 1 has SSL_CERT_FILE, the japps uvicorn subprocess has no CA vars at all. The exchange's urllib calls to the external Keycloak URL (which advertises itself even in-cluster) therefore fail verification, get_conda_envs swallows the empty result, and the dropdown silently vanishes.

Same gap class as #182 (singleuser/app pods, fixed by #102 / released in 0.1.1) — the japps service is one more process that needs the CA and doesn't get it. Note the trust-bundle feature from #102 does not cover it either: _setup_trust_bundle wires spawner pods, not the japps service env.

Suggested fix

In 02-jhub-apps.py, forward the CA vars alongside the client secret, e.g.:

for var in ("SSL_CERT_FILE", "CURL_CA_BUNDLE", "REQUESTS_CA_BUNDLE"):
    val = os.environ.get(var)
    if val:
        svc.setdefault("environment", {})[var] = val

(or, once trust-bundle-enabled is on, point the japps env at the merged bundle path the same way _setup_trust_bundle does for pods).

Workaround

A late hub.extraConfig entry appending SSL_CERT_FILE to the japps service environment (runs after 02-jhub-apps.py registers the service) — deployed successfully as openteams-ai/NIC-argocd-tyler-dev#12.

Environment: nebari-data-science-pack 0.1.1, jhub-apps 2026.5.1rc1, k3s/Hetzner, org CA distributed via trust-manager.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read 02-jhub-apps.py and compare its service environment setup with _setup_trust_bundle; then inspect get_nebi_environments in 03-nebi-envs.py to understand the failing token exchange. Forward the available CA trust variables to the japps service and verify that the Software Environment dropdown populates on a private-CA deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.