kangwonlee / kangwonlee/nmisp

CI: profile nightly image build (3-4x slower than 2023.09)

Open
#370 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
12
Forks
76
PR merge metrics
No merged PRs in 30d

Description

The `nightly` variant of `build-test-image.yml` consistently takes substantially longer than `2023.09`. Observed on commit 593c05fd:

| Variant | Build time |
|---|---|
| `build 2023.09` | 6m56s |
| `build nightly` | ~24m (gated the whole `CI` workflow at 42m total) |

Since the nightly image is rebuilt on every change to `tests/environment.*.yml`, `docker/Dockerfile.nmisp-test`, or the workflow itself, this gating cost is paid often.

## Suggested investigation

1. Compare `tests/environment.2023.09.yml` and `tests/environment.nightly.yml` — likely the nightly env pulls heavier ML deps (TensorFlow / PyTorch nightly wheels are large).
2. Check whether `docker/build-push-action`'s GHA cache (`cache-from: type=gha`, `cache-to: type=gha,mode=max`) is actually hitting on these builds — if cache is invalidated by every Dockerfile change the cache provides little value.
3. Consider splitting the heavy ML install into its own Dockerfile stage so it caches independently from the rest of the env.

## Why

The nightly build dominates the total wall-clock of any CI change that touches the Dockerfile/env. Halving it would meaningfully tighten the iteration loop on container/CI changes.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.