Queued job auto-cancelled ~1s after run start on self-hosted runner pool (no actor, no dispatch)
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Queued job auto-cancelled ~1s after run start on self-hosted runner pool (no actor, no dispatch)
Summary
A job that is never dispatched to a runner is cancelled by GitHub about 1
second after the run is created. The cancelled job shows conclusion: cancelled, runner_id: 0, an empty steps array, canceled_by: null,
and conclusion_reason: null. It never touches a runner. The rest of the
run usually continues and every other job succeeds, but the run-level
conclusion ends cancelled, not failed.
Deterministic: 8 of 8 ci.yml runs on main since 2026-09-07
01:39 UTC show the kill, on push,and schedule triggers, across three
ci.yml revisions (including after we changed concurrency groups,and after
our two workarounds merged) Re-running does not help: two of the observed
runs are attempt-2 runs,and the same job cancelled again about 1-3s
after the re-run was created.
Reproduction (observed, 8/8)
Repo: jperdomo/gamma-origin, workflow ci.yml, job version-lockstep
(mostly( or detect-changes. The pool gamma-build is self-hosted, all
jobs runs-on: gamma-build.
| run id | event | attempt | cancelled jobs | running jobs at death |
|---|---|---|---|---|
| 34073748195 | push | 2 | detect-changes (attempt 2, 02:15:24) | 0 |
| 34127994494 | schedule | 1 | version-lockstep (13:32:53) + 16 of 17 audit-deps legs (13:33:06) | 0 at every death |
| 34132649526 | push | 1 | version-lockstep (14:23:05) | 0 |
| 34157082412 | push | 1 | version-lockstep (19:50:04) | 0 |
| 34159056779 | push | 2 | version-lockstep (20:42:30, attempt 2) | 0 |
| 34164864812 | push | 1 | version-lockstep (21:55:40) | 0 |
In run 34164864812 a second job, warm-buildcache (app: also ended
cancelled (22:02:31) but it had been dispatched (runner_id=7,
steps=12( and died while finishing a different shape, post-dispatch,
not the queued kill this report covers.
What we checked,and ruled out
-
Same-run sibling concurrency race. We previously had all jobs share
oneconcurrencygroup withcancel-in-progress: true. We moved to
per-job groups (ci-${{ github.job }}-${{ github.ref }}, PR #2692( and
the auto-cancel continued unchanged, so the cancel is not caused by our
concurrency configuration -
Newer run superseding. At each cancel instant there is NO other
ci.yml run in progress or queued. A whole-day audit shows exactly one
ci.yml run per incident SHA -
Human or bot cancel.
canceled_byis null in the job payload; no
user or app actor appears. Nothing else in the window calls the cancel API
(ourqueued-run-sweepcron runs withactions: readand cannot cancel) -
Re-run. Re-runs reproduce: attempt-2 runs cancelled the same
job again about 1-3s after re-run creation. A humangh run cancel
cannot explain that (it would need to know the attempt was re-created
within seconds)** -
Runner availability. Other jobs on the same pool dispatch,and run
normally in the same run (build-imagesmatrix,audit-deps,
update-tagsall succeed) At each queued-kill instant zero jobs were
running against the pool`s 14 registered runners -
GitHub-internal queued-cancel is the only model left. Discriminating
cells:the victim is the first-queued job, invariant signature, recursion
in re-runs,recursion across concurrency changes,and zero running jobs at
every death. A known bug class matches: self-hosted jobs stuck or
cancelled atrunner_id=0with an online,idle runner (actions/runner
#3478, #3609;community #186811, #186208).
Expected behavior
A queued job should stay queued until a runner is free, then dispatch,or,
if cancelled, the cancel should carry a reason/actor (conclusion_reason,
canceled_by) so automation can distinguish it from a deliberate cancel.
Actual behavior
The job is marked cancelled about 0-1s after run creation, before any
runner accepts it, with no reason,and no actor. There is currently no
reliable field on the job/run API to distinguish this internal cancel from a
human gh run cancel (our workaround compares started_at - run.created_at)
Impact
The cancelled job is a release-critical assertion; a downstream release gate
had to special-case the exact signature,and our CI now pages on it. Any repo
using a self-hosted pool for a small early job can hit this,and misread it as a
deliberate cancel
Environment
- Actions runner: self-hosted, Linux, pool
gamma-build
Questions for the team
- Is there a known issue with queued-job cancellation on self-hosted pools
during peak queue times? - Is there a field (now or planned( that records an internal cancel reason, so
automation can distinguish it from a user cancel without timing heuristics? - Can we get the Actions queue/dispatch audit log for job 101873798915 (run
34164864812,version-lockstep) from GitHub Support? A concurrency cancel
would name the triggering run_id;a run cancel would show a run-level event;
an internal bug shows no actor,and no reason,the expected outcome here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ci.yml workflow, especially the version-lockstep and detect-changes jobs on the gamma-build self-hosted pool, and inspect the job payload for run 34164864812 and job 101873798915. Compare queued cancellation events with the documented runner and concurrency behavior; done means identifying the cancellation cause or obtaining a reliable reason/actor field that distinguishes it from deliberate cancellation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, linux
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100