aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

perf(ci): shard the CDK jest suite (jest --shard) — DEFERRED, follow-up to #363

Open
#675 0 comments 0 reactions 0 assignees View on GitHub
ci-cd enhancement
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

Parent: #363 (build-performance umbrella). Spun from #363's DoD by the resolve-open-issues orchestrator; **needs maintainer `approved` before implementation (ADR-003)**.

> **⚠️ Re-framed 2026-07-29 after measured review of PR #364 (@theagenticguy).** This lever was originally filed as "the biggest remaining win." **Measurement shows it is currently net-neutral-to-slower and should be DEFERRED behind cheaper levers.** The design below is kept because it is the part people get wrong; the *priority* is what changed.

## Status: DEFERRED (documented design, do not implement yet)

## Why deferred — the numbers moved
The original ~298s `//cdk:test` figure was stale. After PR #371 (disable Lambda bundling in CDK unit-test synths) the suite is **~125s** (136 suites) on the 4-core `merge_group` runner (runs `30412723686` / `30409171662` / `30400147506`, 2026-07-28/29). Substituting today's inputs into the doc's own `slice + fixed_overhead` formula (per-job overhead ~120–220s, and caches currently **miss on every run**):

| Shards | Slice | + overhead | Shard wall |
|---|---|---|---|
| 1 (today) | ~125s | n/a | whole build ~148–192s |
| 2 | ~63s | 120–220s | ~183–283s |
| 4 | ~31s | 120–220s | ~151–251s |

A 4-way shard lands **at or above the current entire build** — before counting the aggregate job's own overhead, extra runner minutes, and coverage-merge complexity. Two further reasons it cannot help yet:
- **Synth is the next binding constraint** (`//cdk:synth:quiet` finishes ~142s), so sharding `//cdk:test` below ~142s buys nothing until synth is also addressed.
- **Caches miss on every `merge_group` run** (`node_modules=miss, venv=miss, jest=miss`) — fixing cache hit-rate is cheaper and *lowers this lever's own crossover point*.

## Do these FIRST (they dominate sharding on current numbers)
1. **#685** — investigate the `Free Disk Space` step (47-67% of pre-build overhead; the largest addressable cost). Cutting it drops fixed_overhead ~120-220s toward ~60-100s.
2. **#677** — bump default runner (vertical scaling: `jest maxWorkers` scales with cores, no per-job overhead duplication). Strictly better than sharding until the largest single runner saturates.
3. **#678** — path-filtered builds (skip `//cdk:test` on non-CDK PRs entirely).
4. **#684** — investigate `yarn install --check-files` (install cost is cache-independent; fixing cache hit-rate saves ~zero — this is NOT a lead lever).

## Revisit trigger
Reconsider sharding **only when `//cdk:test` serial time exceeds ~250s** on the then-current runner AND vertical scaling (#677) is exhausted AND caches are warm — i.e. one runner's cores can no longer hold the needed parallelism, or the suite exhausts a runner's memory/disk. At that point horizontal fan-out becomes the sole remaining lever.

## Design (kept for when the trigger fires — these are the fragile parts)
- Matrix job runs N shards; the **aggregate job must keep the `compute_type: [agentcore]` matrix so its check-run name stays `build (agentcore)`** — the required context in ruleset `14980587`. A job named plain `build` emits check `build`, which does NOT match the required `build (agentcore)` context and **deadlocks the merge queue** (the failure #327 + build.yml's header warn about). This is a REQUIRED-CONTEXT change, not workflow-only.
- Adding a `shard` dimension yields check names `build (agentcore, 1)`, `build (agentcore, 2)`, … — do NOT mark individual shards required (multiplies required contexts).
- **Aggregate gate must use `always()` + a step-level check, NOT a job-level `if: contains(...)`** — `contains()` is not a status function, so `if: ${{ contains(needs.*.result,'failure') }}` silently resolves to `success() && contains(...)` (a contradiction) and the job never runs, admitting failing shards. Correct form:
```yaml
build:
needs: [build-shard]
if: ${{ always() }}
steps:
- name: Gate on shard results
if: ${{ contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') }}
run: exit 1
```
- Merge per-shard coverage before threshold enforcement so `coverageThreshold` still holds on the combined result.
- Run self-mutation / drift checks **once** in the aggregate job, not per shard.

## Secondary (non-wall-clock) benefits, if near the crossover anyway
- Retry only the failing shard instead of the whole suite; isolate a flaky suite's blast radius.

Refs #363

Contributor guide

Open the contributing guide

Research direction

Read build.yml and the parent issue #363, then review the cited merge_group runs and the documented overhead, synth, and cache measurements. Do not implement while the issue is deferred; it is ready for reconsideration only after the stated timing trigger, vertical scaling, and warm-cache conditions are met and a maintainer approves it under ADR-003.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
build-system, ci-cd, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.