CI: tiered container split — design decision before implementation (slim-glibc vs Alpine-musl for the lean tier)
- Dominant language
- Jupyter Notebook
- Stars
- 12
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
## Goal
Most notebooks in this repo don't need TensorFlow / PyTorch / torchvision. Currently the container CI uses one image with a 4.2 GB conda env regardless. Splitting into a lean tier (most notebooks) + an ML tier (the ~4 TF/PyTorch notebooks) should yield meaningful CI time + image-size savings.
This issue is for the **design decision**, not the implementation — there are real tradeoffs worth settling before the rewrite.
## Design space — two candidate base images for the lean tier
| Option | Base | Lean image (est.) | Wheel availability | Build cost |
|---|---|---|---|---|
| A | `python:3.X-slim-bookworm` (glibc, Debian) | ~150–250 MB | full PyPI manylinux — `pip install` of any test dep works | none, official images |
| B | `ghcr.io/kangwonlee/edu-scipy:v0.4.0` (Alpine, musl) | 147 MB | musllinux only — TF/PyTorch/torchvision lack musllinux wheels | none for lean tier; affects future flexibility |
The 4.2 GB conda env is the dominant cost in the current single-tier setup — **most of the size win comes from dropping conda for the non-ML notebooks, not from Alpine specifically**. Slim-glibc captures most of that win without musl complications.
## Constraints worth naming explicitly
1. **No official TF/PyTorch musllinux wheels** — verified 2026-04-21. Building from source is technically possible but heavy: TF needs Bazel (4–6 h, 10–30 GB peak disk), PyTorch needs CMake (1–3 h, 8–15 GB peak disk). GitHub Actions free runners cap at 6 h wall-clock — TF won't fit reliably. Building ourselves can be revisited when we get access to higher-caliber build hardware.
2. **Maintenance burden of custom wheels** — every upstream release = rebuild; every CVE in a transitive dep = rebuild. Becomes a project of its own.
3. **Student reproducibility** — students run notebooks on Colab / Windows / macOS, never on Alpine. Custom musllinux wheels solve a CI-only problem at classroom-scale ongoing cost.
4. **Existing prior work on Alpine** — `ghcr.io/kangwonlee/edu-scipy:v0.4.0` (Alpine, 147 MB) was built 2026-04-12 and is already published. Sunk cost; not by itself a reason to commit to the Alpine path.
5. **Routing mechanism** — already exists in the codebase: `TEST_IPYNB_IGNORE_FOLDER` + per-notebook matrix. Either option uses this unchanged.
6. **The ML tier stays glibc either way** — the ~4 ML notebooks need TF/PyTorch and aren't moving off conda+miniforge regardless of the lean-tier choice.
## Suggested investigation order
1. **Pick the lean base image** (slim-glibc vs Alpine-musl) — load-bearing call. Default-favor slim-glibc unless musl gives a measured benefit beyond the ~50–100 MB image-size delta. Reopen if a future ML-tier dep grows or if the lean tier itself needs something musl-native.
2. **Specify the tier split in the workflow** — which folders go to which tier; concretize the matrix. *Illustrative only* (folder names need verification before commit): `00` / `10` / `15` / `20` / `28` / `30` / `35` / `40` / `60` → lean; ML demos and any TF/PyTorch notebooks → ml.
3. **Build + publish** `nmisp-test-lean` (whichever base) alongside existing `nmisp-test-ml`.
4. **Stage the rollout** — lean-tier CI green on a feature branch before flipping main.
## Related
- #370 — profile nightly image build (3–4× slower than 2023.09). Tactical; this issue is the architectural followup. Together they bound the image-size pain.
- Local verification harness exists at `~/Dev/sync-assignments/docker-sim/docker-sim02/`.
## Why
Long-horizon architectural win. Lean-tier image-pull drops from 4.2 GB to a few hundred MB; the bulk of CI minutes follow. Iteration loop on non-ML changes tightens. The work is bounded — design call (this issue) plus one workflow rewrite — not open-ended.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.