InsightSoftwareConsortium / InsightSoftwareConsortium/ITK

COMP: CI GitHub Actions concurrency bottleneck

Open
#6,074 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.7k
Forks
748
Avg merge
1d 1h
Merged PRs (30d)
64

Description

## Problem

The InsightSoftwareConsortium GitHub org (Team plan) has a **5 concurrent macOS job limit shared across all 162 repositories**. ITK alone has 5-7 macOS jobs per PR across 3 workflows, and the 59 remote module repos each add 4 macOS jobs per PR. A routine CI infrastructure update (e.g., bumping ITKRemoteModuleBuildTestPackageAction from v5.4.5 to v5.4.6) generates **226 macOS jobs** for those 5 slots — queue times exceeded **28 hours** this week.

This is not a transient spike. It's structural: macOS demand scales linearly with open PRs while the slot limit is fixed.

### Concurrency limits (Team plan)

| Resource | Limit | Scope |
|---|---|---|
| Total concurrent jobs | 60 | Per-org, all repos |
| **macOS concurrent jobs** | **5** | **Per-org, all repos** |
| Linux/Windows concurrent jobs | ~55 | Remainder after macOS |

Source: [GitHub Actions limits](https://docs.github.com/en/actions/reference/actions-limits). Limits are per-organization, not per-repo. The macOS cap can be increased via GitHub Support ticket.

### Current macOS job demand

ITK macOS jobs per PR

| Workflow | Runner | Jobs | What it tests |
|---|---|---|---|
| ITK.Arm64 / x86_64-rosetta | macos-15 | 1 | C++ x86_64 via Rosetta |
| ITK.Arm64 / Python | macos-15 | 1 | Python wrapping (ARM) |
| ITK.Pixi / macos | macos-15 | 1 | C++ with conda-forge toolchain |
| ITK.macOS.Arm64 (release-5.4 only) | macos-14 | 2 | C++ + Python (legacy workflow) |
| Azure/macOS | macos-15 | 1 | C++ shared libs + examples (not counted against GH limit) |
| Azure/macOS.Python | macos-15 | 1 | Python wrapping (not counted against GH limit) |

**Per PR on main: 3 GH Actions macOS jobs**
**Per PR on release-5.4: 5 GH Actions macOS jobs**

Remote module macOS jobs per PR

Each remote module using the shared ITKRemoteModuleBuildTestPackageAction triggers:

| Job type | Count | macOS slots |
|---|---|---|
| C++ (ubuntu, windows, macos-intel, macos-arm) | 4 | 2 |
| Python wheels (linux×2, linux-arm×2, macos-arm×2, windows×2) | 8 | 2 |
| Lint | 1 | 0 |
| **Total per module** | **13** | **4** |

**59 modules × 4 macOS jobs = 226 macOS jobs** when all have open PRs.

Measured queue times (2026-04-15)

| Repo | Queued macOS jobs | Oldest in queue |
|---|---|---|
| ITK | 20 runs (Arm64 + Pixi + macOS.Arm64) | 27.3h |
| ITKMontage | 7 | 28.6h |
| ITKFixedPointInverseDisplacementField | 3 | 15.0h |
| ITKElastix | 1 | 16.5h |

Two ITK jobs marked "in_progress" for 27+ hours were actually hung, not running.

## Proposed fixes

### 1. Request macOS slot increase from GitHub Support (immediate)

The Team plan allows custom limits via support ticket. Increasing from 5 to 15-20 would cut queue times by 3-4×. This is the single highest-impact change.

### 2. Add `concurrency` groups to all workflows (immediate)

Force-pushes to a PR branch currently queue new runs without cancelling the old ones. Adding cancellation prevents superseded runs from holding macOS slots:

```yaml
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
```

`arm.yml` and `pixi.yml` already have this. `macos-arm.yml` (release-5.4) does **not**.

### 3. Reduce macOS CI for remote module PRs (medium-term)

Most remote modules are pure C++ algorithm filters. For PR validation, Linux-only CI is sufficient to catch compilation and test failures. The full platform matrix (including macOS) should run only on merges to main/master.

The shared action could add an `os-list` default that excludes macOS for `pull_request` events:

```yaml
# In the remote module's workflow:
os-list: ${{ github.event_name == 'pull_request' && '["ubuntu-22.04"]' || '["ubuntu-22.04", "windows-2022", "macos-15-intel", "macos-15"]' }}
```

This cuts per-module macOS jobs from 4 to 0 for PRs.

### 4. Reduce Python wheel matrix for PRs (medium-term)

Building cp310 + cp311 × 4 platforms = 8 wheel jobs per PR. For validation, 1 version on 1 platform is sufficient. Full matrix on merge only.

### 5. Consider dropping ITK.Arm64/x86_64-rosetta (low priority)

Rosetta x86_64 testing on macOS-15 is a niche concern. Azure/macOS already covers macOS C++ builds. Removing this saves 1 macOS slot per ITK PR.

### 6. Delete `macos-arm.yml` from release-5.4 (low priority)

This adds 2 macOS-14 jobs per release-5.4 PR that duplicate coverage already provided by `arm.yml` (macos-15) and Azure/macOS.

## Related

- #6060 — Remote module consolidation (in-tree migration reduces CI pressure structurally)
- #6051 — v5.4.6 backport tracking (triggered the congestion)
- ITKRemoteModuleBuildTestPackageAction — shared CI action used by all 59 modules

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.