nodejs / nodejs/node-core-utils

checkGitHubCI blocks indefinitely on orphaned check suites stuck in QUEUED

Open
#1,160 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
313
Forks
137
Avg merge
2d 8h
Merged PRs (30d)
24

Description

Problem

checkGitHubCI() in pr_checker.js blocks landing with ✘ GitHub CI is still running when any GitHub Actions check suite has status !== 'COMPLETED'. However, GitHub occasionally creates check suites that remain stuck in QUEUED status with 0 check runs indefinitely — they are never dispatched and will never complete.

This has blocked at least two recent PRs on nodejs/node:

  • #64991Test macOS suite stuck 3+ days, landed manually
  • #64830 — same Test macOS suite stuck, landed manually

In both cases, the orphaned suite:

  • Has status: QUEUED, conclusion: null
  • Has 0 check runs (never started executing)
  • Has a valid workflowRun (Test macOS, workflow ID 858952)
  • Was created at the same time as sibling suites that completed fine
  • Never gets re-triggered by request-ci because GitHub sees it as already "queued"
  • The only workaround is a collaborator manually running git node land --yes

The commit-queue.sh pre-check (gh pr checks | grep pending) passes fine because it evaluates individual check runs (all completed), not suite containers.

Data

PR #64991, SHA 94a47d6f15b2:

  • 25 GitHub Actions suites total
  • 24 completed (success/skipped)
  • 1 orphaned: suite 84102428609, Test macOS, QUEUED, 0 check runs, created 2026-08-05T13:19:48Z

No newer run of Test macOS exists for this SHA — request-ci does not re-trigger a workflow GitHub considers already "queued."

Proposed fix

Consider a suite as orphaned and skip it if:

  • status !== 'COMPLETED'
  • checkRuns.nodes.length === 0 (nothing has ever executed inside it)
  • Suite age exceeds a configurable threshold (suggesting 3 hours since createdAt — legitimate suites dispatch runs within seconds)
if (status !== 'COMPLETED') {
  const runCount = checkRuns?.nodes?.length ?? 0;
  const ageMs = Date.now() - new Date(createdAt).getTime();
  if (runCount === 0 && ageMs > ORPHANED_SUITE_TIMEOUT) {
    cli.warn(`Ignoring orphaned check suite with no runs (status: ${status}, age: ${Math.round(ageMs / 3600000)}h)`);
    continue;
  }
  pendingJobs.push({ app: app.slug, status, conclusion });
  continue;
}

GraphQL change: add createdAt to the checkSuites query in PRCommits.gql.

Trade-offs

What this fixes: Orphaned suites no longer block the commit queue forever.

Risk: If someone adds commit-queue before request-ci has finished triggering all workflows, a suite that should have run but hasn't started yet (0 runs, within the threshold) could age past the timeout and get skipped — landing the PR without that test suite's results.

However, the current workaround (git node land --yes) already bypasses ALL checks — including real failures. A targeted skip of empty aged-out suites is strictly safer than the current manual workaround.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with checkGitHubCI() in pr_checker.js and inspect the checkSuites query in PRCommits.gql, then compare its suite data with the commit-queue.sh pre-check behavior. Define and validate the configurable age threshold so empty, aged QUEUED suites no longer block landing while active suites remain pending; confirm the warning identifies skipped orphaned suites.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, graphql, javascript
Domain
ci-cd, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.