Reproducibility: what would it take to rebuild a published lecture exactly?
Nobody has claimed this yet.
- Dominant language
- Shell
- Stars
- 0
- Forks
- 1
- Avg merge
- 32m
- Merged PRs (30d)
- 3
Description
From QuantEcon/meta#52 (2021, closed as answered): can containers give us reproducible QuantEcon environments? Answered for our CI here; this issue holds the rest.
1. No environment a reader or collaborator can reproduce. Images are internal — an anonymous pull of ghcr.io/quantecon/quantecon returns UNAUTHORIZED — so readers get Colab.
2. Our own builds are not reproducible either.
- Consumers float on a mutable tag: builds emit a commit-SHA tag beside
:latest(type=sha,prefix={{branch}}-→main-<short-sha>,.github/workflows/build-containers.yml:63,:105), but every consumer and doc pins:latest(README.md:108,docs/CONTAINER-GUIDE.md:9-10,setup-environment/README.md:34-35); neither is stable, since the weekly rebuild (cron: '0 2 * * 1',:11) re-pushes both at the same commit. The SHA names the recipe commit, not the image; only the digest is durable and nothing records it — the publish manifest carries name/tag/commit/timestamp/size, no container identity (publish-gh-pages/action.yml:126-135). - Recipes float too:
FROM ubuntu:24.04, tag not digest (containers/quantecon/Dockerfile:1,containers/quantecon-build/Dockerfile:1); unpinned TeXLive/systemapt-get install(containers/quantecon/Dockerfile:8); unpinnedquantecon-book-networksin both env files (containers/quantecon/environment.yml:25,containers/quantecon-build/environment.yml:74); lean image's Jupyter Book stack floor-pinned>=(containers/quantecon-build/environment.yml:65-73). Only the Miniconda installer is fetched under a checksum (containers/quantecon/Dockerfile:77-78), so date- or digest-addressing a past image still would not make it rebuildable. - Unversioned AMI: lecture repo GPU workflows name the RunsOn AMI bare,
image=quantecon_ubuntu2404, no digest or version. - Unpinned build-time installs:
pip install -U "jax[cuda13]"in bothlecture-python.mystandlecture-jax, pluspip install numpyroin the former.
Only the first two are owned here, but all four compose.
Identify (which environment built this publish?) is cheap and already homed: #30's field list includes container tag/digest, PLAN.md:82 names container digest capture. Rebuild (from the recipe) needs a base-image digest pin plus an apt pin set, and must not be assumed by it.
Any answer must be additive: floating is deliberate — exact pins on action refs stranded lecture-dp three releases behind (PLAN.md:33), and the proposed fix there is a staging tag plus a release gate (#135), not a repin of consumers.
- Record the building commit SHA in
/etc/quantecon-container, today a bare marker line, an untaggedimage=ghcr.io/quantecon/quantecon, andbuild_date(containers/quantecon/Dockerfile:99-101). Makes containers self-identifying; #30 needs it regardless.
QuantEcon/meta#349 (pixi lockfiles) and QuantEcon/meta#165 (uv) cover the conda and pip layer, both scoped to maintainer/CI tooling; the system and reader layers are untracked. Not urgent, not necessarily one piece of work — filed to give the question a public home.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with containers/quantecon/Dockerfile, containers/quantecon-build/Dockerfile, and .github/workflows/build-containers.yml, then review PLAN.md:82 and the referenced consumer files. The immediately actionable scope is the /etc/quantecon-container marker, but the broader issue requires defining how identification and recipe reproducibility should coexist with floating tags; done criteria are not yet consolidated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, shell
- Domain
- build-system, ci-cd, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100