CI: infra failures and pre-existing nightly failures found during #3235 validation

Tracking issue for problems surfaced while validating #3235 (megatron dependency
simplification) against its base commit `bc382dc45`.

**None of these are caused by #3235.** That PR introduced **zero regressions** across the full
nightly suite on H100 and GB200 (128 tests). This issue exists to track the pre-existing and
infrastructure failures found along the way, per the triage in that PR's description.

---

## 1. Corrupt git object cache on a CI runner working dir (blocking)

A runner working directory has a corrupt `nemo-ci` object store. Every job scheduled onto it
dies in `get_sources` within ~15s:

```
Reinitialized existing Git repository in ...//0/dl/JoC/nemo-ci/.git/
fatal: missing blob object '4611bc350167d6730c9e978013552758924176f6'
error: remote did not send all necessary objects
```

Same blob SHA every time. It hit **6 jobs across 4 pipelines** over ~14 hours.

Small, targeted pipelines are hit disproportionately: they tend to land on the same low-numbered
concurrency slot, which is the corrupt one. Broad pipelines spread across other slots and mostly
succeed — which is why the full nightly ran fine but single-test re-runs kept dying.

**Fix:** wipe that runner's `dl/JoC/nemo-ci` checkout so it re-clones.

**Impact:** this is the sole reason `llm_grpo_gemma3_1b_it_1n4g_fsdp2tp1` has no valid
branch-side result (see §3).

## 2. `scheduler_failure` burst on GB200

In the first GB200 nightly, **9 of 28 jobs** reported `failed` with `failure_reason=scheduler_failure`
and a duration of ~0.03s — they never started. All 9 were re-run; **8 passed**, the 9th failed for
an unrelated pre-existing reason.

This matters for triage hygiene: at face value that run looked like "13 failures", when only 4 were
real. Anyone reading raw GB200 pass/fail counts should filter on `failure_reason` first.

## 3. `llm_grpo_gemma3_1b_it_1n4g_fsdp2tp1` — unexplained `ModuleNotFoundError` (GB200)

First branch run failed with:

```
ModuleNotFoundError: No module named 'nemo_rl.distributed.collectives'
```

raised from the cluster's `code_snapshots//nemo_rl/...` copy. `nemo_rl/distributed/collectives.py`
exists at HEAD, and #3235 touches nothing under `nemo_rl/distributed/`, so an incomplete code
snapshot is the leading hypothesis — but this was never confirmed, because three further attempts
were all consumed by §1.

Passes on main. **Needs one clean GB200 run to close out.**

## 4. `llm_grpo_llama3_2_1b_instruct_1n8g_megatron_generation_tq_mooncake` — intermittent teardown hang (H100)

Failed once on the branch at 9650s, passed on retry at 8779s, passes on main at 8631s → **flaky**.

The failure mode is worth a look regardless. Training completes normally:

```
Max number of steps has been reached, stopping training early
```

then the job hangs in Ray teardown and is eventually cancelled:

```
Policy.__del__ -> worker_group.shutdown -> ray.get(futures, timeout=timeout)
```

A hang *after* the work is done wastes a full wall-clock slot and reports as a test failure.
Possibly related: `gb200_L1_Functional_Tests_Megatron_3` in GitHub Actions hung with no output
for ~5.5h and was killed at the 6h job limit.

## 5. Pre-existing nightly failures at `bc382dc45`

These fail identically on branch and main — already broken on main, listed for visibility.

**H100 (11)**
- `llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3`
- `llm_grpo_llama3_1_8b_instruct_1n8g_megatron_fp8_rollouts_v3_tq_simple`
- `llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e`
- `llm_grpo_llama3_1_8b_instruct_2n8g_megatron_fp8_e2e_tq_mooncake`
- `llm_grpo_qwen3_1_7b_6n8g_megatron_super_rlvr`
- `llm_grpo_qwen3_30ba3b_4n8g_megatron_qa_nvfp4`
- `llm_grpo_qwen3_30ba3b_4n8g_megatron_zmq_deltaweight_noncolocated`
- `llm_grpo_qwen3_30ba3b_thinking_swe1_16n8g_megatron_cp2_r3_async_gym`
- `llm_grpo_qwen3_5_35ba3b_2n8g_megatron_ep16tp2cp2`
- `llm_grpo_qwen3_8b_base_1n8g_fp8_kvcache_megatron`
- `vlm_vlm_grpo_qwen3_omni_30ba3b_audiomcq_4n8g_megatron_v1`

**GB200 (4)**
- `llm_dpo_nanov3_30B3AB_1n4g_fsdp4ep4_automodel`
- `llm_grpo_nanov3_30ba3b_4n4g_megatron_qa_nvfp4_w4a16_real`
- `llm_grpo_nemotron3_super_120BA12B_8n4g_megatron`
- `llm_grpo_qwen3_30ba3b_4n4g_megatron_qa_nvfp4_w4a4_real`

## 6. `pull_request` checks silently skipped on conflicting PRs

When a PR is `mergeable=CONFLICTING`, GitHub cannot compute `refs/pull/N/merge`, so **every
`pull_request`-event workflow is skipped with no red signal** — including `Copyright check`,
`Secrets detector`, and `Lockfile freshness`. `pull_request_target` workflows and the copy-pr-bot
`push` pipeline are unaffected, so the PR still looks checked.

A conflicting PR therefore merges its copyright/secrets scanning silently unenforced once the
conflict is resolved and someone hits merge without re-running. Worth considering a required
check that fails loudly on conflict.

---

Full per-test table with job and pipeline IDs: #3235.