ci: neither analysis workflow has ever completed a scheduled run — GEMM pins its checkout to a deleted branch, RCCL's docker login is unauthorized

Problem

gemm-sweep-analysis.yml and rccl-warp-speed-analysis.yml run on a 0 0 */2 * * cron and publish results to ROCm/aorta-report. Neither has ever completed a scheduled run successfully — not once in 129 (GEMM) and 115 (RCCL) scheduled attempts. The last green run of either was a pull_request run on 2026-01-10, about 7.5 months ago, and aorta-report has received nothing from these workflows since 2026-01-10.

They are broken for two different reasons.

GEMM dies on its first step. Line 42 pins the checkout to a personal branch that no longer exists:

      - name: Checkout AORTA repository
        uses: actions/checkout@v5
        with:
          repository: ROCm/aorta
          ref : prosenj_gh_action
          path: aorta

repos/ROCm/aorta/branches/prosenj_gh_action returns 404, and the ref is absent from git ls-remote on ROCm/aorta. The branch was the head of #55 ("Enabling RCCL and GEMM sweep test on GH action"), merged to main on 2026-01-12 as a8f01d7 and deleted afterwards — so the workflow shipped with its own development-time pin intact and has been unable to check out since the day its PR was tidied up.

Worth knowing before you go looking for it: the key is written ref :, with a space before the colon. YAML accepts that (the space is not part of the key), so the line is invisible to a rg 'ref:' sweep and reads as normal at a glance. rg 'ref\s*:' finds it.

rccl-warp-speed-analysis.yml does not have this defect — it checks out ref: ${{ github.ref }} (~51), which is why it gets much further before failing.

RCCL dies at the Docker registry login. The step is named Build Docker container, but the build never starts; docker login is rejected:

Docker Compose version v2.27.1
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get "https://registry-1.docker.io/v2/": unauthorized: incorrect username or password
##[error]Process completed with exit code 1.

That is docker login -u rocmshared -p ${{ secrets.ROCM_SHARED_KEY }} (~113) — the ROCM_SHARED_KEY secret is no longer valid for the rocmshared account. docker compose build and everything downstream never execute, and publish-results is skipped.

These are two blockers, not one, and GEMM has both. GEMM's own Build Docker container step (~57) runs the identical docker login -u rocmshared -p ${{ secrets.ROCM_SHARED_KEY }}. So repairing the ref moves GEMM forward exactly one step and lands it on the same unauthorized login. Neither fix alone revives either workflow.

Evidence

All figures below are from the complete run history via the Actions API on 2026-08-25, not a truncated window.

	gemm-sweep-analysis.yml	rccl-warp-speed-analysis.yml
Total runs	144 (first 2025-12-15T17:10:52Z)	123 (first 2026-01-07T13:38:38Z)
schedule / cancelled	107	93
schedule / failure	22	22
schedule / success	0	0
pull_request / success	3 (all on prosenj_gh_action)	2 (all on prosenj_gh_action)
Last green run	2026-01-09T19:09:54Z — 20862674941	2026-01-10T14:15:23Z — 20879579801

Every success either workflow has ever had was a pull_request run on the branch that has since been deleted. No run on main, by any event, has ever succeeded.

Two eras, and the first one was grey rather than red

Era 1 — queued, never picked up (GEMM 2025-12-17 → 2026-06-29, 100 consecutive cancelled; RCCL 2026-01-13 → 2026-06-29, 86 consecutive cancelled). Mechanism verified on GEMM run 26197916167 (2026-05-21): created_at 00:25:40Z, updated_at 2026-05-22T00:25:42Z — exactly 24 hours — with runner_name: "" and steps: []. The job was queued, never assigned a runner, and GitHub cancelled it at the 24-hour queue limit. Both workflows request the bare label runs-on: self-hosted (GEMM ~34, RCCL ~43), whereas sanitizers-nightly.yml ~37 targets [self-hosted, gpu] and carries a comment that the MI350 host is labelled self-hosted, gpu, mi350x. I did not verify the runner label sets, so treat "the bare label matched no online runner" as the likely explanation rather than an established one; the 24-hour timeout with no runner assigned is established.

Era 2 — picked up, fails in seconds. First scheduled failure for both is 2026-07-01: GEMM 28484875851, RCCL 28485163081. Seven more cancelled each between 2026-07-07 and 2026-07-19 (runner availability still intermittent), then 19 consecutive failure each, 2026-07-21 through 2026-08-25.

This split matters for the "why did nobody notice" question below: for the first six months these runs concluded cancelled, which is grey in the Actions UI and does not trip failure notifications. The period with the most consecutive bad runs was also the least visible one.

Current failure points, from the logs

GEMM run 32792112062 (2026-08-25), runner smci350-rck-g03-f16-12, step Checkout AORTA repository:

[command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules \
  --depth=1 origin +refs/heads/prosenj_gh_action*:refs/remotes/origin/prosenj_gh_action* \
  +refs/tags/prosenj_gh_action*:refs/tags/prosenj_gh_action*
The process '/usr/bin/git' failed with exit code 1

— three attempts (checkout's own retry), then ##[error]Process completed with exit code 14. Note the runner did pick the job up, so the label problem of era 1 is not the current blocker.

RCCL run 32792315456 (2026-08-25): the unauthorized login quoted above, at 00:07:59 — 39 seconds into the job.

Is ref: main the right fix?

At the script-interface level, yes — verified. What the scripts run inside is the open question.

Verified against origin/main today:

• All 15 paths either workflow invokes exist on main: docker/docker-compose.rocm70_9-1.yaml, docker/rccl_test/docker-compose.rocm70_9-1.yaml, requirements.txt, train.py, config/single_node/gemm_overlap_comm.yaml, scripts/gemm_analysis/{run_train_various_channels.sh,run_tracelens_analysis.sh,analyze_gemm_reports.py,plot_gemm_variance.py,enhance_gemm_variance_with_timestamps.py,gemm_report_with_collective_overlap.py,process_gpu_timeline.py,process_comms.py}, scripts/tracelens_single_config/{run_rccl_warp_speed_comparison.sh,run_full_analysis.py}.
• The scripts have not drifted at all since the workflow was written. git diff a8f01d7 origin/main -- scripts/gemm_analysis scripts/tracelens_single_config is empty — byte-identical between the #55 merge commit and main today. Repointing the checkout at main therefore hands the workflow exactly the scripts it was authored against.
• Every flag the workflow passes is still accepted, checked in main's copies: run_train_various_channels.sh --output-dir/--channels/--threads/--config; analyze_gemm_reports.py --base-path/--threads/--channels/--ranks/--top-k; plot_gemm_variance.py --csv-path/--output-dir; enhance_gemm_variance_with_timestamps.py --input-csv/--base-path; gemm_report_with_collective_overlap.py --input-csv/--tracelens-path; process_gpu_timeline.py / process_comms.py --sweep-dir; run_full_analysis.py --baseline/--test/--output/--all/--skip-tracelens/--compare-all-runs.

So the ref change does not silently change which analysis code runs, and it does not break an interface. The one-line change is safe in that specific sense, and that is the only sense I can establish.

Named unknown. What the scripts execute against has moved since the merge. Changed between a8f01d7 and origin/main:

 config/single_node/gemm_overlap_comm.yaml       | 18 ++++++-------
 docker/docker-compose.rocm70_9-1.yaml           | 20 ++++++++++++++
 docker/rccl_test/Dockerfile.rocm70_9-1          | 36 ++++++++++++++++++++++---
 docker/rccl_test/docker-compose.rocm70_9-1.yaml |  3 ++-
 requirements.txt                                | 24 ++++++++++-------

That is the default --config training YAML, both compose files, the RCCL Dockerfile and the pinned dependencies — i.e. the container and the training configuration, not the analysis code. Neither workflow has ever executed against any of this, because nothing has got past docker login since July and nothing at all has been green since January. Whether the sweep still runs end to end is therefore genuinely unknown and will be discovered by the first successful run, not by review. This is inference, not verification: I could not test it read-only and without GPU runners.

The practical consequence for whoever picks this up: treat ref as one of at least three things to fix (checkout ref, registry credential, runner labels), and treat the first green run as the start of the investigation rather than the end of it. A shape worth considering, rather than a prescription: GEMM's explicit repository: + ref: override could simply be dropped so it checks out the triggering ref the way RCCL already does — which removes the class of defect instead of re-pinning it to a different branch name.

Secondary observation — nothing in the repo surfaces a permanently-broken scheduled workflow

Clearly secondary to the report above; recorded here at the reporter's request rather than as a separate issue.

The sharper form of the finding is not that the repo has no alerting — it is that it has exactly one mechanism, and these two workflows are outside it while nightly-eval is inside it. Verified:

• No workflow in the repo uses if: failure(). rg 'failure\(\)' .github/workflows/ returns no matches at all.
• No Slack / Teams / webhook / email step anywhere, by grep across .github/workflows/.
• The one mechanism is scripts/ci/alert_issue.py, invoked from eval-reusable.yml ~125-134 under if: always() && inputs.do_alert. It files, updates or closes a single issue labelled nightly-regression — a real, auto-managed repo label. nightly-eval.yml ~37 opts in (do_alert: true); bump-validate.yml ~31 deliberately opts out (do_alert: false, "PRs must not file/close the nightly regression issue"). Both analysis workflows have no equivalent, and neither does sanitizers-nightly.yml.
• Even that mechanism would not have caught this. It is content-driven: it decides from FAIL verdicts inside gpu-nightly-results.json, and the calling step no-ops with no results JSON; skipping alert when the run produced nothing. A job that dies at checkout or at docker login produces no results file. So the repo can alert on "the eval ran and regressed" and has no signal for "the workflow could not run at all" — which is precisely the failure mode here.
• sanitizers-nightly.yml sits between the two: no issue filing, but it publishes status.json (healthy, conclusion, run_url) to its data branch and renders a stale banner (~327-355), so a dead nightly is at least visible on the dashboard. Neither analysis workflow publishes any liveness signal.

Combined with the era-1 cancelled conclusions, that fully accounts for six months of silence: grey runs, no failure() hook, no webhook, and the only alerting path gated behind a results file that was never produced.

Acceptance criteria

• GEMM's checkout resolves without a hardcoded personal branch, and a scheduled run gets past Checkout AORTA repository.
• docker login succeeds in both workflows — a refreshed ROCM_SHARED_KEY, a different credential, or removal of the login if the images no longer need authentication.
• runs-on targets a label set a live runner actually carries, so a scheduled run cannot sit queued for 24 hours and conclude cancelled.
• One scheduled run of each completes green end to end and publishes to aorta-report — the first ever for GEMM's schedule, and the first since 2026-01-10 for RCCL.
• That run confirms the workflows still work against today's docker/** compose files, docker/rccl_test/Dockerfile.rocm70_9-1, requirements.txt and config/single_node/gemm_overlap_comm.yaml — asserted by a green run, not assumed from review.
• #395 is resolved before either workflow is revived, or the revival carries the unique-destination change with it.
• Secondary: a decision is recorded on whether a scheduled workflow that is repeatedly failure or cancelled should raise any signal, and whether these two opt into it.

Files

• .github/workflows/gemm-sweep-analysis.yml — cron ~4-6, runs-on: self-hosted ~34, ref : prosenj_gh_action ~42, docker login ~57, aorta-report checkout ~169-175, publish ~177-192
• .github/workflows/rccl-warp-speed-analysis.yml — cron ~4-6, runs-on: self-hosted ~43, ref: ${{ github.ref }} ~51, docker login ~113, publish-results job ~343-381
• .github/workflows/eval-reusable.yml — the repo's only alerting step ~125-134
• .github/workflows/nightly-eval.yml — do_alert: true ~37
• .github/workflows/bump-validate.yml — do_alert: false ~31
• .github/workflows/sanitizers-nightly.yml — runner labels + label comment ~34-37, status.json liveness signal ~327-355
• scripts/ci/alert_issue.py — content-driven alerting, nightly-regression label ~26

Notes

• This is why #395 is still only latent. That issue's same-day collision in aorta-report/<date>/ cannot fire while neither workflow can reach its publish step — which is the argument for fixing #395 before these workflows are revived rather than after. Reviving them first makes the first two same-day dispatches the thing that discovers the collision, on published data.
• The GEMM ref defect was found while investigating #395 and is what made that issue's run-history evidence thin: with zero successful publishes there is nothing to observe empirically.
• #396 is the third finding from the same sweep (date-only commit subjects on the ci-results branch) — related only by provenance, no shared cause.
• Provenance of the pin: #55, merged 2026-01-12 as a8f01d7, head branch prosenj_gh_action deleted after merge.

Start with .github/workflows/gemm-sweep-analysis.yml and .github/workflows/rccl-warp-speed-analysis.yml, then inspect the cited failed runs and the runner labels used by sanitizers-nightly.yml. Verify the checkout, runner, and registry-login behavior in both workflows, and confirm that scheduled runs reach the analysis and result-publishing steps without introducing script or configuration drift.