Lightning-AI / Lightning-AI/lightning-thunder
Flaky test `thunder/tests/test_dynamo.py::test_reports_benchmark`
Open
@kiya00 is already working on this.
Since Aug 20, 2025.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
Note: If you have a model or program that is not supported yet but should be, please use the program coverage template.
🐛 Bug
It runs out of time in CI sometimes, but can't reproduce locally
thunder/dynamo/report.py:884: in run_benchmark
return timer_fn.time("compiled_fn(*inputs)", globals={"compiled_fn": compiled_fn, "inputs": inputs})
thunder/dynamo/benchmark_utils.py:358: in time
measurement = t.adaptive_autorange(
/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/timer.py:437: in adaptive_autorange
times = self._threaded_measurement_loop(
/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/timer.py:299: in _threaded_measurement_loop
can_stop = stop_hook(times)
/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/timer.py:431: in stop_hook
return common.Measurement(
/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/common.py:186: in meets_confidence
return self.iqr / self.median < threshold
/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/common.py:126: in iqr
self._lazy_init()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <[ValueError('cannot convert float NaN to integer') raised in repr()] Measurement object at 0x7fa914f203a0>
def _lazy_init(self) -> None:
if self.raw_times and not self._sorted_times:
self._sorted_times = tuple(sorted(self.times))
_sorted_times = torch.tensor(self._sorted_times, dtype=torch.float64)
> self._median = _sorted_times.quantile(.5).item()
E Failed: Timeout (>240.0s) from pytest-timeout.
/usr/local/lib/python3.10/dist-packages/torch/utils/benchmark/utils/common.py:167: Failed
You can reproduce this example by temporarily adding @reproduce_failure('6.136.9', b'AEEB') as a decorator on your test case
= 1 failed, 2020 passed, 317 skipped, 22 xfailed, 7 xpassed, 58639 warnings in 501.63s (0:08:21) =
##[error]Bash exited with code '1'.
Finishing: Testing: main
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.