ci: reduce Acceptance Tests runtime from ~1h40m (runner queue saturation + per-job rebuilds)
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 274
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 103
Description
## Problem
The `Acceptance Tests` workflow currently takes ~1h40m. Analysis of [run 27106300746](https://github.com/owncloud/ocis/actions/runs/27106300746) shows the dominant cost is **not test execution time** — it's runner queue saturation and redundant per-job work.
### Timeline (measured)
| Phase | Start offset | Duration | Note |
|---|---|---|---|
| Workflow queued → `detect-changes` starts | +0m | **9m37s** | Runner queue wait |
| `detect-changes` runs | +9m37s | 58s | |
| `detect-changes` done → `build-and-test` starts | +10m35s | **14m23s** | Runner queue wait |
| `build-and-test` runs | +24m58s | 8m57s | Builds oCIS, lint, unit tests |
| Gate open → first test job starts | +33m55s | 6m11s | Runner queue |
| Gate open → last test job starts | +33m55s | **56m46s** | 55 jobs competing for runners |
| Last test job finishes | +101m17s | 626s actual test | |
**Total ~101 minutes. Runner queue wait alone accounts for 65+ minutes.**
---
## Issue 1: Every test job rebuilds oCIS from source
`run-github.py` line 381 calls `make -C ocis build` inside every one of the 55 test matrix jobs. The `build-and-test` job already builds the binary — then discards it. Each downstream job re-runs:
- `actions/setup-go` (~21s)
- `apt-get install libssl-dev libnghttp2-dev … libvips-dev` (~36s, even on cache hit)
- `make -C ocis build` + `make -C tests/ociswrapper build`
- `make ci-node-generate` (IDP/web assets)
- `composer install` (PHP deps)
**Fix:** Upload the built binaries and generated assets as an artifact from `build-and-test`, download them in each test job. This removes the need for `setup-go`, `setup-node`/pnpm, libcurl/libvips, and the compile step from all 55 jobs.
```yaml
# In build-and-test, after building:
- uses: actions/upload-artifact@v4
with:
name: ocis-binaries
path: |
ocis/bin/ocis
tests/ociswrapper/bin/ociswrapper
services/web/assets/
services/idp/assets/
retention-days: 1
# In each test job, replace the build steps with:
- uses: actions/download-artifact@v4
with:
name: ocis-binaries
- run: chmod +x ocis/bin/ocis tests/ociswrapper/bin/ociswrapper
```
PHP composer deps could also be uploaded/restored, or kept since composer is fast.
---
## Issue 2: 55 jobs launched simultaneously — pool exhaustion
After `build-and-test` finishes, all 55 test jobs are queued at once. The runner pool can't supply them simultaneously, so jobs are delayed by up to **56 minutes** waiting for a runner slot. This is the single biggest contributor to total runtime.
### Fix A: Tier execution (fail-fast cheap → expensive)
Add a `tier-1-smoke` job group with the fast protocol/smoke tests (cs3api=55s, litmus=102s, wopi-builtin=86s, wopi-cs3=104s, cliCommands=668s). Use `fail-fast: true` so a failure in tier-1 cancels its siblings immediately. Make `tier-2-full` depend on `tier-1-smoke` — if tier-1 fails, the 49-job expensive wave never launches.
```yaml
tier-1-smoke:
needs: [build-and-test]
strategy:
fail-fast: true
matrix:
suite: [cs3api, litmus, wopi-builtin, wopi-cs3, "cliCommands,apiServiceAvailability"]
tier-2-full:
needs: [build-and-test, tier-1-smoke]
strategy:
fail-fast: false
matrix:
suite: [... all remaining suites ...]
```
On a broken PR, this stops the run after ~12 minutes instead of 100 minutes.
### Fix B: Align matrix groups with the existing Python logical groups
`run-github.py` already defines logical test groups that share services (email, tika, antivirus, etc.). The matrix currently uses 32 individual suite entries for `local-api-tests`, but the runner script already handles comma-separated suite lists. Aligning the matrix with the existing groups reduces concurrent runner demand by ~60%:
| Current (32 single-suite jobs) | Proposed (grouped) | Combined duration |
|---|---|---|
| apiGraph + apiDownloads + apiGraphGroup | 1 job: `apiGraph,apiDownloads,apiGraphGroup` (matches `groupAndSearch1`) | ~617s |
| apiSpacesDavOperation + apiDownloads + apiAsyncUpload + apiDepthInfinity + apiArchiver + apiActivities | 2 jobs (matches `davOperations`, split for time) | ~600s each |
| apiSharingNgShares + apiReshare + apiSharingNgPermissions | 1-2 jobs (matches `sharingNg1`) | ~1345s → 2 jobs |
| apiSharingNgDriveLinkShare + apiSharingNgItemLinkShare + apiSharingNgLinkShareManagement | 1 job (matches `sharingNgLinkShare`) | ~978s |
This reduces from 32 to ~14-15 local-api runner slots while keeping per-job runtime under ~700s.
---
## Issue 3: `detect-changes` queue wait (~10 minutes for a 58-second job)
`detect-changes` waits ~9m37s for a runner to run a 58-second script. One option is to use a `paths-filter` action directly in the workflow trigger condition, or use `concurrency` groups more aggressively to avoid needing a separate job for this.
---
## Measured job durations (for grouping reference)
Top 10 slowest test jobs (total wall time including setup):
| Job | Total | Actual test step |
|---|---|---|
| apiSharingNgShares | 834s | 752s |
| apiGraphUser | 796s | 736s |
| apiSpacesShares | 789s | 730s |
| coreApiWebdavProperties | 776s | 664s |
| coreApiSharePublicLink2 | 762s | 677s |
| apiSpaces | 683s | 603s |
| coreApiShareOps (grouped) | 674s | 568s |
| cliCommands,apiServiceAvailability | 668s | 629s |
| apiSharingNgItemInvitation | 648s | 591s |
| apiSearchContent | 626s | 540s |
Setup overhead (checkout + setup-go + setup-node + pnpm + setup-php + libcurl) per job: **~55-90s** — this overhead disappears if binaries are pre-built.
---
## Projected impact
| Fix | Complexity | Success path savings | Failure path savings |
|---|---|---|---|
| Upload/download binary artifact | Medium | ~2min per job setup eliminated; removes need for Go/Node toolchain in test jobs | Same |
| Align matrix with existing Python groups (32→14 jobs) | Low | Reduces peak runner demand ~40%; last job starts ~25m earlier | Big |
| Tier 1/2 split (fail-fast) | Low | Minimal on success | Stops 49-job wave on broken PRs |
| Fix `detect-changes` queue | Low | ~10m off the front | Same |
**Combined realistic estimate: 35-45 minutes off the 1h40m runtime** on the success path, mostly from runner queue reduction. Failure path could be ~30min total instead of ~100min.
Contributor guide
Assessment
This issue has not been assessed yet.