actions / actions/runner

Queued job auto-cancelled ~1s after run start on self-hosted runner pool (no actor, no dispatch)

Open
#4,684 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Queued job auto-cancelled ~1s after run start on self-hosted runner pool (no actor, no dispatch)

Summary

A job that is never dispatched to a runner is cancelled by GitHub about 1
second after the run is created. The cancelled job shows conclusion: cancelled, runner_id: 0, an empty steps array, canceled_by: null,
and conclusion_reason: null. It never touches a runner. The rest of the
run usually continues and every other job succeeds, but the run-level
conclusion ends cancelled, not failed.

Deterministic: 8 of 8 ci.yml runs on main since 2026-09-07
01:39 UTC show the kill, on push,and schedule triggers, across three
ci.yml revisions (including after we changed concurrency groups,and after
our two workarounds merged) Re-running does not help: two of the observed
runs are attempt-2 runs,and the same job cancelled again about 1-3s
after the re-run was created.

Reproduction (observed, 8/8)

Repo: jperdomo/gamma-origin, workflow ci.yml, job version-lockstep
(mostly( or detect-changes. The pool gamma-build is self-hosted, all
jobs runs-on: gamma-build.

run id event attempt cancelled jobs running jobs at death
34073748195 push 2 detect-changes (attempt 2, 02:15:24) 0
34127994494 schedule 1 version-lockstep (13:32:53) + 16 of 17 audit-deps legs (13:33:06) 0 at every death
34132649526 push 1 version-lockstep (14:23:05) 0
34157082412 push 1 version-lockstep (19:50:04) 0
34159056779 push 2 version-lockstep (20:42:30, attempt 2) 0
34164864812 push 1 version-lockstep (21:55:40) 0

In run 34164864812 a second job, warm-buildcache (app: also ended
cancelled (22:02:31) but it had been dispatched (runner_id=7,
steps=12( and died while finishing a different shape, post-dispatch,
not the queued kill this report covers.

What we checked,and ruled out

  1. Same-run sibling concurrency race. We previously had all jobs share
    one concurrency group with cancel-in-progress: true. We moved to
    per-job groups (ci-${{ github.job }}-${{ github.ref }}, PR #2692( and
    the auto-cancel continued unchanged, so the cancel is not caused by our
    concurrency configuration

  2. Newer run superseding. At each cancel instant there is NO other
    ci.yml run in progress or queued. A whole-day audit shows exactly one
    ci.yml run per incident SHA

  3. Human or bot cancel. canceled_by is null in the job payload; no
    user or app actor appears. Nothing else in the window calls the cancel API
    (our queued-run-sweep cron runs with actions: read and cannot cancel)

  4. Re-run. Re-runs reproduce: attempt-2 runs cancelled the same
    job again about 1-3s after re-run creation. A human gh run cancel
    cannot explain that (it would need to know the attempt was re-created
    within seconds)**

  5. Runner availability. Other jobs on the same pool dispatch,and run
    normally in the same run (build-images matrix, audit-deps,
    update-tags all succeed) At each queued-kill instant zero jobs were
    running against the pool`s 14 registered runners

  6. GitHub-internal queued-cancel is the only model left. Discriminating
    cells:the victim is the first-queued job, invariant signature, recursion
    in re-runs,recursion across concurrency changes,and zero running jobs at
    every death. A known bug class matches: self-hosted jobs stuck or
    cancelled at runner_id=0 with an online,idle runner (actions/runner
    #3478, #3609;community #186811, #186208).

Expected behavior

A queued job should stay queued until a runner is free, then dispatch,or,
if cancelled, the cancel should carry a reason/actor (conclusion_reason,
canceled_by) so automation can distinguish it from a deliberate cancel.

Actual behavior

The job is marked cancelled about 0-1s after run creation, before any
runner accepts it, with no reason,and no actor. There is currently no
reliable field on the job/run API to distinguish this internal cancel from a
human gh run cancel (our workaround compares started_at - run.created_at)

Impact

The cancelled job is a release-critical assertion; a downstream release gate
had to special-case the exact signature,and our CI now pages on it. Any repo
using a self-hosted pool for a small early job can hit this,and misread it as a
deliberate cancel

Environment

  • Actions runner: self-hosted, Linux, pool gamma-build

Questions for the team

  • Is there a known issue with queued-job cancellation on self-hosted pools
    during peak queue times?
  • Is there a field (now or planned( that records an internal cancel reason, so
    automation can distinguish it from a user cancel without timing heuristics?
  • Can we get the Actions queue/dispatch audit log for job 101873798915 (run
    34164864812, version-lockstep) from GitHub Support? A concurrency cancel
    would name the triggering run_id;a run cancel would show a run-level event;
    an internal bug shows no actor,and no reason,the expected outcome here.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ci.yml workflow, especially the version-lockstep and detect-changes jobs on the gamma-build self-hosted pool, and inspect the job payload for run 34164864812 and job 101873798915. Compare queued cancellation events with the documented runner and concurrency behavior; done means identifying the cancellation cause or obtaining a reliable reason/actor field that distinguishes it from deliberate cancellation.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, linux
Domain
ci-cd, devops
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.