Nebi workspaces written under $HOME on user PVC can fill the volume; collides badly with jupyter_scheduler

Summary

A singleuser pod CrashLoopBackOff'd with OSError: [Errno 28] No space left on device because the home PVC (20Gi) hit 100% full. Investigation showed the dominant disk cost is the combination of two design choices in this pack:

1. Nebi workspaces (with their full pixi envs) are placed under $HOME on the user PVC via the env vars in images/nebi/jupyter_server_config.py:

   "NEBI_STORAGE_WORKSPACES_DIR": os.path.join(
       os.environ.get("XDG_DATA_HOME", os.path.expanduser("~/.local/share")),
       "nebi", "workspaces",
   )
   

   A single CUDA/PyTorch-class workspace is easily 5–10 GB. Multiple workspaces fill 20 GB fast.

2. jupyter_scheduler recursively copies the notebook's working directory into a staging area, with no exclude defaults and no hardlink/reflink awareness (jupyter_scheduler/utils.py:copy_directory). When $HOME is the notebook dir, every scheduled job duplicates .local/share/nebi/workspaces/** into .local/share/jupyter/scheduler_staging_area/<uuid>/. The copy is larger than the original, because the original is hardlinked into the rattler cache but shutil.copytree materializes full files.

Observed footprint on the affected PVC (20 Gi, 100% full)

Path	Size	Notes
~/.local/share/jupyter/scheduler_staging_area/<uuid>/.local/share/nebi/	10.9 GB	Orphaned scheduler staging copy, ~3 weeks old
~/.local/share/nebi/workspaces/<workspace>/.pixi	7.2 GB	Real pixi env (hardlinked to rattler cache)
~/.cache/rattler/cache/uv-cache	6.0 GB	uv wheel cache (hardlinked)
~/.cache/rattler/cache/pkgs	1.8 GB	Conda pkg cache (hardlinked)

Apparent total > actual disk usage because of hardlinks, but the staging-area copy de-duplicates against rattler and is fully materialized — that one directory alone was over half the PVC.

Notably, the sibling nebari-nebi-pack/templates/deployment.yaml already points NEBI_STORAGE_WORKSPACES_DIR at a dedicated mount — so the standalone server gets this right. It's the singleuser image config that's the outlier.

Why this isn't only a one-user problem

Anyone with a non-trivial pixi workspace will hit it, and jupyter_scheduler amplifies it on each run. Related: #55 notes the scheduler service isn't fully wired up yet, but the staging directory in this incident is dated, so it has been runnable at some point.

Options for fixing (need discussion — not opening a PR yet)

Three independent levers; doing any one would have prevented the incident.

A. Move Nebi workspaces off \$HOME

Override NEBI_STORAGE_WORKSPACES_DIR (and NEBI_DATABASE_DSN) to a path outside \$HOME. Two flavors:

• A1. Separate PVC, like the nebari-nebi-pack server chart already does. Cleanest separation between user notebooks and heavy artifacts. Requires Helm wiring for a second PVC + mount on singleuser pods.
• A2. Same PVC, non-\$HOME path (e.g. /srv/nebi/workspaces, mount-shared with HOME). Solves the scheduler-copy problem without changing the storage topology. Cheaper to ship.

B. Trim what jupyter_scheduler stages

Subclass jupyter_scheduler.scheduler.Scheduler to exclude heavy dotdirs, wired via c.SchedulerApp.scheduler_class in the singleuser jupyter_server_config.py. Smallest, lowest-risk change. Two sub-options for the policy:

• B1. Targeted denylist (recommended): exclude known-heavy paths only — .local, .cache, .pixi, .conda, .rattler, .ipynb_checkpoints. Small dotfiles like .env, .gitignore still come along.
• B2. Blanket dotfile exclude + allowlist file: skip all dotfiles by default, honor a .scheduler-include manifest at the notebook dir for opt-in.

C. Upstream fixes (separate, longer-horizon)

• File against jupyter-server/jupyter-scheduler to (1) expose exclude_files as a Traitlet, (2) ship sensible default excludes, (3) optionally support os.link/reflink to avoid duplicating hardlinked trees.
• File against nebari-dev/nebi to document the NEBI_STORAGE_WORKSPACES_DIR knob as required deployment config when running inside a JupyterHub pod with a HOME PVC.

My take (open to challenge)

Combining A2 + B1 is probably the right immediate landing zone for this pack: move workspaces out of \$HOME so scheduler stops sweeping them, and trim scheduler staging defensively so unrelated heavy dotdirs don't bite us in the future. A1 (separate PVC) is the more principled answer but adds Helm/storage complexity that may not be justified yet.

Immediate workaround for affected users without a code change: bump the PVC (storage class supports allowVolumeExpansion: true) and rm -rf ~/.local/share/jupyter/scheduler_staging_area/*.

Priority?

I don't know if this is high priority — wanted to write it up before the context goes cold. Happy to do any of A/B/C as a PR once the team picks a direction.

Start with images/nebi/jupyter_server_config.py and compare its workspace settings with nebari-nebi-pack/templates/deployment.yaml. Review jupyter_scheduler/utils.py, especially copy_directory, to understand the staging behavior. Done means implementing the team-selected mitigation and verifying that workspace data is not needlessly copied into scheduler staging or allowed to exhaust the home PVC.