🐛 CI failure: JET launcher killed with exit 137 after GPT-583M inference tests pass

**Describe the bug**

Six GPT-583M dynamic-inference CI jobs are marked failed even though their final child GPU workloads succeeded and the downloaded test artifacts show successful inference/validation. The parent `launch_jet_workload.py` process is killed with exit code 137 after downloading the workload assets, before replaying the test logs.

This is a post-test launcher failure, not an observed CUDA runtime error or golden-value mismatch in these six jobs. Updating their goldens will not address the launcher termination.

@NVIDIA/mcore-oncall

**Failing run**

- PR: [#6991 — chore: Update base image to 26.08](https://github.com/NVIDIA/Megatron-LM/pull/6991)
- Tested commit: [`a4d622074ccb990aaa4598923cec4c998774ea11`](https://github.com/NVIDIA/Megatron-LM/commit/a4d622074ccb990aaa4598923cec4c998774ea11)
- Observed: September 10, 2026; platform `dev_dgx_h100`.
- Internal CI pipeline ID: `67130833`. Internal CI links are intentionally omitted.

| Parent job ID | Affected test recipe |
| --- | --- |
| `433441346` | `gpt_dynamic_inference_tp1_pp1_583m_chunked_prefill_cuda_graphs` |
| `433441347` | `gpt_dynamic_inference_tp1_pp1_583m_cuda_graphs_fp8_logitsmatch` |
| `433441353` | `gpt_dynamic_inference_tp1_pp1_583m_prefix_caching_chunked_prefill_cuda_graphs` |
| `433441355` | `gpt_dynamic_inference_tp1_pp1_583m_prefix_caching_cuda_graphs` |
| `433441368` | `gpt_dynamic_inference_tp2_pp2_583m_cuda_graphs` |
| `433441370` | `gpt_dynamic_inference_tp2_pp2_583m_prefix_caching_cuda_graphs` |

**Error**

The parent job reports the Python launcher process as `Killed`, followed by:

```text
ERROR: Job failed: command terminated with exit code 137
```

Evidence distinguishing this from a test failure:

- All six final child GPU workloads succeeded; downloaded rank-0 exit artifacts are `0`.
- The prefix-caching and TP2/PP2 jobs (`433441353`, `433441355`, `433441368`, `433441370`) each contain five successful inference executions and five passing `test_inference_pipeline` checks (`1 passed`).
- The chunked-prefill job (`433441346`) also has explicit passing inference validation in its successful replacement child workload; an earlier child launch failed before producing test logs.
- Parent termination occurs after asset downloads, before the normal main-log replay. There is no Python traceback establishing why the launcher was killed.

**Steps/Code to reproduce bug**

Observed in the normal JET-backed functional CI workflow at the tested commit:

1. Run one of the affected recipes on `dev_dgx_h100` through `tests/test_utils/python_scripts/launch_jet_workload.py`.
2. Allow the child GPU workload and its inference validation to finish.
3. Observe the parent launcher while it downloads and processes the workload logs/artifacts.
4. Compare the successful child status and exit artifacts with the parent's exit code 137.

A minimal deterministic reproduction has not yet been established. Running the pytest validation alone does not exercise the failing parent log-handling path.

**Expected behavior**

The parent launcher should finish processing the logs and report the successful workload result without being killed.

**Additional context**

- Host-memory pressure during log aggregation is a hypothesis, **not a confirmed OOM**. Exit code 137 alone does not establish the source of SIGKILL; runner/container memory counters or OOM events are needed.
- At the tested commit, [`extract_torchrunlogs_to_string`](https://github.com/NVIDIA/Megatron-LM/blob/a4d622074ccb990aaa4598923cec4c998774ea11/tests/test_utils/python_scripts/launch_jet_workload.py#L263) loads per-rank logs using `readlines()`. The [post-download path](https://github.com/NVIDIA/Megatron-LM/blob/a4d622074ccb990aaa4598923cec4c998774ea11/tests/test_utils/python_scripts/launch_jet_workload.py#L506) also loads the main log and builds concatenated log strings. This is a relevant area to measure, not a proven fault location.
- Suggested investigation: capture launcher peak RSS and runner/container termination details, correlate them with downloaded log sizes, and assess whether log processing can be streamed or bounded.
- The decode-only throughput failure and optimized-SwiGLU log-probability mismatch are separate failures and are not included in this issue.
- No assignee was selected: the latest change to the first affected recipe on `main` was authored by the `svcnemo-autobot` automation account.
- Triaged using the `mcore-create-issue` skill.

Start in tests/test_utils/python_scripts/launch_jet_workload.py, especially extract_torchrunlogs_to_string and the post-download path around line 506. Reproduce an affected JET-backed recipe if possible, capture launcher peak RSS and runner/container termination details, and compare them with downloaded log sizes. Done means the parent processes successful child results and reports them without exit 137, with the termination cause supported by evidence.

Written by the indexing model from the issue text.