InsightSoftwareConsortium / InsightSoftwareConsortium/ITK
COMP: CI GitHub Actions concurrency bottleneck
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 748
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 64
Description
## Problem
The InsightSoftwareConsortium GitHub org (Team plan) has a **5 concurrent macOS job limit shared across all 162 repositories**. ITK alone has 5-7 macOS jobs per PR across 3 workflows, and the 59 remote module repos each add 4 macOS jobs per PR. A routine CI infrastructure update (e.g., bumping ITKRemoteModuleBuildTestPackageAction from v5.4.5 to v5.4.6) generates **226 macOS jobs** for those 5 slots — queue times exceeded **28 hours** this week.
This is not a transient spike. It's structural: macOS demand scales linearly with open PRs while the slot limit is fixed.
### Concurrency limits (Team plan)
| Resource | Limit | Scope |
|---|---|---|
| Total concurrent jobs | 60 | Per-org, all repos |
| **macOS concurrent jobs** | **5** | **Per-org, all repos** |
| Linux/Windows concurrent jobs | ~55 | Remainder after macOS |
Source: [GitHub Actions limits](https://docs.github.com/en/actions/reference/actions-limits). Limits are per-organization, not per-repo. The macOS cap can be increased via GitHub Support ticket.
### Current macOS job demand
ITK macOS jobs per PR
| Workflow | Runner | Jobs | What it tests |
|---|---|---|---|
| ITK.Arm64 / x86_64-rosetta | macos-15 | 1 | C++ x86_64 via Rosetta |
| ITK.Arm64 / Python | macos-15 | 1 | Python wrapping (ARM) |
| ITK.Pixi / macos | macos-15 | 1 | C++ with conda-forge toolchain |
| ITK.macOS.Arm64 (release-5.4 only) | macos-14 | 2 | C++ + Python (legacy workflow) |
| Azure/macOS | macos-15 | 1 | C++ shared libs + examples (not counted against GH limit) |
| Azure/macOS.Python | macos-15 | 1 | Python wrapping (not counted against GH limit) |
**Per PR on main: 3 GH Actions macOS jobs**
**Per PR on release-5.4: 5 GH Actions macOS jobs**
Remote module macOS jobs per PR
Each remote module using the shared ITKRemoteModuleBuildTestPackageAction triggers:
| Job type | Count | macOS slots |
|---|---|---|
| C++ (ubuntu, windows, macos-intel, macos-arm) | 4 | 2 |
| Python wheels (linux×2, linux-arm×2, macos-arm×2, windows×2) | 8 | 2 |
| Lint | 1 | 0 |
| **Total per module** | **13** | **4** |
**59 modules × 4 macOS jobs = 226 macOS jobs** when all have open PRs.
Measured queue times (2026-04-15)
| Repo | Queued macOS jobs | Oldest in queue |
|---|---|---|
| ITK | 20 runs (Arm64 + Pixi + macOS.Arm64) | 27.3h |
| ITKMontage | 7 | 28.6h |
| ITKFixedPointInverseDisplacementField | 3 | 15.0h |
| ITKElastix | 1 | 16.5h |
Two ITK jobs marked "in_progress" for 27+ hours were actually hung, not running.
## Proposed fixes
### 1. Request macOS slot increase from GitHub Support (immediate)
The Team plan allows custom limits via support ticket. Increasing from 5 to 15-20 would cut queue times by 3-4×. This is the single highest-impact change.
### 2. Add `concurrency` groups to all workflows (immediate)
Force-pushes to a PR branch currently queue new runs without cancelling the old ones. Adding cancellation prevents superseded runs from holding macOS slots:
```yaml
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
```
`arm.yml` and `pixi.yml` already have this. `macos-arm.yml` (release-5.4) does **not**.
### 3. Reduce macOS CI for remote module PRs (medium-term)
Most remote modules are pure C++ algorithm filters. For PR validation, Linux-only CI is sufficient to catch compilation and test failures. The full platform matrix (including macOS) should run only on merges to main/master.
The shared action could add an `os-list` default that excludes macOS for `pull_request` events:
```yaml
# In the remote module's workflow:
os-list: ${{ github.event_name == 'pull_request' && '["ubuntu-22.04"]' || '["ubuntu-22.04", "windows-2022", "macos-15-intel", "macos-15"]' }}
```
This cuts per-module macOS jobs from 4 to 0 for PRs.
### 4. Reduce Python wheel matrix for PRs (medium-term)
Building cp310 + cp311 × 4 platforms = 8 wheel jobs per PR. For validation, 1 version on 1 platform is sufficient. Full matrix on merge only.
### 5. Consider dropping ITK.Arm64/x86_64-rosetta (low priority)
Rosetta x86_64 testing on macOS-15 is a niche concern. Azure/macOS already covers macOS C++ builds. Removing this saves 1 macOS slot per ITK PR.
### 6. Delete `macos-arm.yml` from release-5.4 (low priority)
This adds 2 macOS-14 jobs per release-5.4 PR that duplicate coverage already provided by `arm.yml` (macos-15) and Azure/macOS.
## Related
- #6060 — Remote module consolidation (in-tree migration reduces CI pressure structurally)
- #6051 — v5.4.6 backport tracking (triggered the congestion)
- ITKRemoteModuleBuildTestPackageAction — shared CI action used by all 59 modules
Contributor guide
Assessment
This issue has not been assessed yet.