NVIDIA / NVIDIA/TensorRT-LLM

[Bug] test_unittests_v2[unittest/_torch/executor] wrapper fails intermittently (~40%) with all inner tests passing — session teardown crash

Open
#16,284 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Pytorch Testing
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Summary

test_unittests_v2[unittest/_torch/executor] fails intermittently in pre-merge single-GPU stages with AssertionError: failure reported in unittests while all ~953 inner executor tests report PASSED in the merged results.

Mechanics (from the wrapper stack): the parallel phase exits non-zero after tests complete, the --lf retry re-runs and passes everything, but the session still exits non-zero — i.e. a crash/failure in session teardown (thread-leak check or native crash at shutdown), not a test failure.

Occurrences (all on unrelated PR #15730, current-main bases)

  • L0_MergeRequest_PR 47040 — B300-PyTorch-2
  • L0_MergeRequest_PR 47191 — DGX_H100-PyTorch-3
  • L0_MergeRequest_PR 47357 — B300-PyTorch-2
  • L0_MergeRequest_PR 47366 — B300-PyTorch-2

That is 4 of ~10 runs — the single most frequent pre-merge blocker we hit while babysitting that PR. The NCCL shutdown fix (#15892) did not eliminate it.

Ask

Deflake the executor unittest group teardown (e.g. identify the leaking/crashing shutdown path, or apply the pytest.mark.threadleak(enabled=False) pattern used elsewhere if it is the threadleak gate).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing test_unittests_v2[unittest/_torch/executor] across repeated single-GPU runs and inspect the parallel phase, --lf retry, and session teardown. Investigate the executor group’s thread-leak check or native shutdown path, comparing the pytest.mark.threadleak(enabled=False) pattern used elsewhere. Done means the wrapper exits successfully without intermittent teardown failures while the inner tests still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.