NVIDIA-NeMo / NVIDIA-NeMo/Gym

ci: prepare and enable a merge queue for main

Open
#3,196 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tracking
Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Problem

main does not currently use a merge queue. A pull request can be green against an earlier base commit and then merge
after main has advanced without its full gate being re-evaluated against the final integration commit. Auto-merge does
not serialize concurrent merges or validate a pull request together with earlier queued changes.

Enabling the repository setting immediately is not sufficient. Workflows that provide required checks must first handle
the merge_group event and reliably report a terminal result for the synthetic queue commit. Otherwise queue entries can
remain blocked waiting for checks that never run.

Goal

Prepare the CI and branch-protection contract for merge-group validation, exercise it safely, and only then require a
merge queue for main.

Work plan

1. Define the merge gate
  • Inventory every check that should block a merge and identify its workflow/app owner.
  • Choose stable, unique required-check names, preferably aggregate checks for sharded or conditional suites.
  • Define expected behavior for code, docs-only, fork, Dependabot, and externally approved CI paths.
  • Ensure skipped or conditionally unnecessary work still produces a definitive successful aggregate check rather
    than leaving a required check pending.
  • Document which checks run before queue entry and which must run again on the merge-group commit.
2. Make CI merge-group aware
  • Add merge_group: { types: [checks_requested] } to every workflow that owns a required queue check, or route the
    event through one dedicated queue-gate workflow.
  • Remove assumptions that github.event.pull_request and its base/head fields are present.
  • Check out and test the merge-group SHA, not the original pull-request head SHA.
  • Update change classification and base-ref selection for merge_group events.
  • Verify reusable workflows, submodule checkout, runner routing, permissions, secrets, and external CI work with
    gh-readonly-queue/* refs.
  • Ensure concurrency keys do not cancel unrelated queue groups or reuse a stale pull-request result.
  • Keep post-merge main CI as defense in depth, while avoiding accidental duplicate publishing or deployment from
    merge-group runs.
3. Validate before enforcement
  • Exercise the configuration on a temporary protected branch or another reversible test target.
  • Queue two compatible pull requests and confirm the second is tested with the first and the latest target branch.
  • Confirm a failing entry does not merge and that a subsequent healthy entry can proceed.
  • Test docs-only, fork, cancelled, superseded, and manually retried entries for permanently pending checks.
  • Record expected check names and a rollback/debugging runbook for maintainers.
4. Enable the merge queue
  • Configure required pull requests and the audited required status checks for main.
  • Enable Require merge queue only after all required checks report successfully on test merge_group runs.
  • Start with conservative build and merge concurrency, then tune using observed queue latency and runner capacity.
  • Verify the configured merge method matches the repository's squash-only policy.
  • Monitor initial queue runs and post-merge main health before considering the rollout complete.

Acceptance criteria

  • Every queued change is tested at the exact merge-group SHA that includes the latest main and any preceding queued
    changes.
  • A required check cannot be satisfied solely by a stale pull-request-head result.
  • Failed, cancelled, and skipped paths always resolve without silently merging or leaving the queue permanently pending.
  • Fork and docs-only pull requests have a documented, tested path through the queue.
  • Only a successful merge-group gate can update main; the existing post-merge CI remains operational.
  • Maintainers have a short runbook for queue failures, bypass policy, and rollback.

Out of scope

Broad CI performance work or unrelated workflow refactors. Those can be tracked separately unless they are required to
make the merge-group gate correct and reliable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inventorying the repository's CI workflow files and required checks, then trace how they respond to pull_request and main events. Add and test merge_group checks using the queue commit SHA, including docs-only and fork paths, before changing main's branch protection. Done means queued changes receive terminal checks, only a successful merge-group gate can update main, and the rollback runbook is recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, github-actions
Domain
ci-cd
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.