nebari-dev / nebari-dev/data-science-pack

Set jupyterhub.cull.maxAge as a backstop for activity-based culling

Open
#228 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
7
Avg merge
1d 20h
Merged PRs (30d)
13

Description

jupyterhub.cull.maxAge is 0 (unset) in values.yaml, so nothing bounds how long a user pod can live. Every culler we run is activity-based, and activity is a signal we don't fully control.

Concretely, the hub culler can't be fixed from inside the pod. CHP updates route activity on any request or reply data including websocket frames:

https://redirect.github.com/jupyterhub/configurable-http-proxy/blob/main/lib/configproxy.js#L649-L656

and the hub folds that into spawner.last_activity with max():

https://redirect.github.com/jupyterhub/jupyterhub/blob/5.5.0/jupyterhub/app.py#L3595-L3602

so any proxied app holding a websocket keeps the hub's view of the pod fresh no matter what the in-pod server reports. That's the VS Code case in #208, but it'll be true of anything else we put behind jupyter-server-proxy.

max_age is the one hub-level lever that doesn't care. It's evaluated in its own branch keyed off started, independent of the inactivity check:

https://redirect.github.com/jupyterhub/jupyterhub-idle-culler/blob/main/jupyterhub_idle_culler/__init__.py#L259-L270

2i2c runs timeout: 1800 / every: 300 / maxAge: 43200 in production:

https://infrastructure.2i2c.org/sre-guide/manage-k8s/culling/

and minrk's guidance in https://redirect.github.com/jupyterhub/jupyterhub-idle-culler/issues/55 is the same shape, short in-pod deadline and a long hub one.

Proposal is to set maxAge to something in the 12h range and document it as a hard cap rather than an idle timeout. Open question is what number actually fits our users. 12h kills a pod mid-session for anyone doing a long training run who isn't touching the browser, so I'd want to know if that's a real pattern here before picking it.

This is independent of #226. That PR fixes what counts as activity, this bounds what happens when the answer is still wrong.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with values.yaml and review how the existing culler settings are configured. Confirm what maxAge value fits users' workloads, then set it as a hard pod lifetime cap and document that it is independent of the idle timeout. Check the resulting configuration against the referenced production settings and culling behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
devops
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.