[CI/CD] Merge queue orphaned workflow runs waste CI resources

Summary

GitHub's merge queue does not automatically cancel workflow runs when they become orphaned due to queue reordering, PR updates, or PR removals. This results in wasted CI resources (runner minutes and concurrency slots) and can delay actual merge operations.

Problem Description

When the merge queue reorders PRs or a PR in the queue gets a new commit, GitHub creates new workflow runs with different merge commit SHAs but does not cancel the now-obsolete older runs. These orphaned runs continue consuming resources until completion, even though their results will be ignored.

Real Example from Today (2026-02-10)

Orphaned Run 1:

• PR #34579, Run ID: 21873979471
• Started: 16:46:40 UTC
• Merge SHA: 1a5716f4
• Duration: ~64 minutes
• Result: ✅ Success
• Why orphaned: PR #34558 merged at 16:47:09, triggering queue recalculation
• New run created: 16:47:15 with different SHA 15764c12
• Wasted resources: 64 minutes of CI time discarded

Orphaned Run 2:

• PR #34579, Run ID: 21874000991
• Started: 16:47:15 UTC
• Merge SHA: 15764c12
• Why orphaned: Queue reordered again at 18:02:38
• New run created: 18:02:38 with different SHA b7512e4d
• Action taken: Manually cancelled to save resources

Orphaned Run 3:

• PR #34581, Run ID: 21876263967
• Started: 17:53:05 UTC
• Merge SHA: 248d66a1
• Why orphaned: Queue reordered at 18:02:38
• New run created: 18:02:38 with different SHA 74c92b57
• Action taken: Manually cancelled

Root Cause

The merge queue creates new runs when:

1. ✅ A PR in the queue merges (removes itself, reorders remaining PRs)
2. ✅ A new PR is added to the queue (batch composition changes)
3. ✅ A PR in the queue gets a new commit (invalidates previous test)
4. ✅ Main branch is updated outside merge queue (requires rebase)

Each of these triggers new merge commit SHAs, but GitHub does not cancel the obsolete runs with old SHAs.

Impact

Resource Waste

• CI Minutes: Orphaned runs complete full test suites (~25-30 min each)
• Concurrency: Occupies runner slots that could serve current runs
• Cost: For large teams with high queue churn, this can be significant

Merge Velocity

• Reduced available concurrency slows down active queue runs
• False sense of progress (orphaned runs show as "in progress")

Example Waste Today

• 3 orphaned runs identified
• Estimated waste: ~2 hours of runner time (if not cancelled)
• 2 runs manually cancelled, saving ~1 hour

Upstream Issue

This is a known GitHub limitation with no official fix:

• GitHub Community Discussion: #137976 - Merge Queue: Github does not cancel workflows in merge queue even when they become irrelevant
• Status: Open since November 2024, no GitHub response
• Community consensus: "fundamental feature gap"

Proposed Solutions

Solution 1: Add merge_queue.destroyed Event (Recommended)

Leverage undocumented GitHub feature to auto-cancel orphaned runs:

name: -2 Merge Group Check

on:
  merge_group:
    types: [checks_requested, destroyed]  # ← Add 'destroyed'

concurrency:
  group: merge-queue-${{ github.event.merge_group.head_ref || github.ref }}
  cancel-in-progress: true

jobs:
  cleanup:
    if: github.event.action == 'destroyed'
    runs-on: ubuntu-latest
    steps:
      - name: Auto-cancel orphaned runs
        run: |
          echo "Merge queue entry destroyed - workflow will auto-cancel via concurrency group"
          
  # Existing test jobs
  test:
    if: github.event.action == 'checks_requested'
    runs-on: ubuntu-latest
    # ... existing job definition

How it works:

• When a PR is removed from queue (merged, failed, reordered), GitHub fires destroyed event
• The destroyed workflow run triggers, then gets cancelled by cancel-in-progress: true
• This cancels other runs in the same concurrency group (same head_ref)

Source: Community-discovered workaround from Discussion #137976

Risks:

• ⚠️ destroyed event is undocumented and may change/break
• ⚠️ Requires testing to ensure it works as expected

Solution 2: Enhanced Concurrency Groups (Fallback)

If destroyed event proves unreliable, use concurrency groups alone:

concurrency:
  group: merge-queue-${{ github.ref }}
  cancel-in-progress: true

Limitations:

• Only cancels runs for the same github.ref
• Does not cancel when different merge commits are created
• Partial solution but better than nothing

Solution 3: Periodic Cleanup Script (Manual)

Create a scheduled workflow to identify and cancel orphaned runs:

name: Cleanup Orphaned Merge Queue Runs

on:
  schedule:
    - cron: '*/15 * * * *'  # Every 15 minutes
  workflow_dispatch:

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - name: Cancel old merge queue runs
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          # Get all in-progress merge queue runs
          gh run list --workflow="cicd_2-merge-queue.yml" \
            --status in_progress \
            --json databaseId,createdAt,headBranch \
            --limit 100 > runs.json
          
          # For each PR, find runs with different SHAs and cancel older ones
          jq -r '.[] | [.databaseId, .createdAt, .headBranch] | @tsv' runs.json | \
          awk '{
            pr = gensub(/.*pr-([0-9]+)-.*/, "\\1", 1, $3)
            sha = gensub(/.*pr-[0-9]+-(.*)/, "\\1", 1, $3)
            key = pr "-" sha
            
            if (key in latest) {
              # Older run found, cancel it
              print "Cancelling orphaned run:", $1
              system("gh run cancel " $1)
            } else {
              latest[key] = $1
            }
          }'

Limitations:

• Requires maintenance
• May hit API rate limits with high activity
• 15-minute delay means some waste still occurs

Solution 4: Third-Party Action (Not Recommended)

Use styfle/cancel-workflow-action:

• ❌ No documented merge queue support
• ❌ Adds external dependency
• ❌ May hit API rate limits

Recommended Implementation Plan

1. Phase 1 (Immediate): Implement destroyed event handler

   • Add to .github/workflows/cicd_2-merge-queue.yml
   • Test with a sample PR to verify auto-cancellation
   • Monitor for any unexpected behavior

2. Phase 2 (Fallback): Add periodic cleanup script

   • Create .github/workflows/cleanup-orphaned-queue-runs.yml
   • Schedule every 15 minutes
   • Acts as safety net if destroyed event fails

3. Phase 3 (Monitoring): Track effectiveness

   • Log when orphaned runs are cancelled
   • Measure CI minute savings
   • Document any issues with destroyed event

Acceptance Criteria

• destroyed event handler added to merge queue workflow
• Concurrency groups properly configured
• Manual cleanup script created as fallback
• Test with sample PR confirms auto-cancellation works
• Document any edge cases or limitations discovered
• Track CI minute savings over 1 week

References

External Issues

• GitHub Community Discussion #137976 - Core issue and destroyed event workaround
• Managing a Merge Queue - GitHub Docs - Official documentation

Analysis

• Today's orphaned runs identified via CI/CD diagnostics investigation
• Manual cancellation saved ~1 hour of runner time (2 runs cancelled)
• Pattern analysis shows queue reordering happens frequently during high activity

---

Priority: Medium-High
Effort: Low (1-2 hours implementation + testing)
Impact: Reduces CI costs and improves merge velocity
Risk: Low (worst case: no change from current state)

Start with .github/workflows/cicd_2-merge-queue.yml and the merge_group event configuration. Check how destroyed events and concurrency groups behave, then review the proposed cleanup workflow and its shell, jq, and awk logic. Done means orphaned runs are cancelled, the fallback is available, and the documented acceptance checks and limitations are covered.