🐛 [Bug] Torch-TensorRT Slower than Onnx TensorRT in Alpamayo-10B
Open
Nobody has claimed this yet.
bug
- Dominant language
- Python
- Stars
- 3k
- Forks
- 410
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 78
Description
Benchmark Result
Per-clip minADE (m)
| # | Clip | ONNX-TRT | Torch-TRT | Δ |
|---|---|---|---|---|
| 1 | 0043b781… |
0.1719 | 0.1721 | +0.0002 |
| 2 | 00ee8960… |
0.9539 | 0.9582 | +0.0043 |
| 3 | 0145f6e0… |
0.3925 | 0.3929 | +0.0004 |
| 4 | 01460b45… |
0.0876 | 0.0874 | −0.0002 |
| 5 | 01cf8186… |
0.8914 | 0.7777 | −0.1137 |
| 6 | 021a4585… |
0.0976 | 0.1003 | +0.0027 |
| 7 | 0347d9f9… |
1.5423 | 1.5467 | +0.0044 |
| 8 | 03aa0b51… |
1.3659 | 1.3488 | −0.0171 |
| 9 | 0455a6e4… |
0.2546 | 0.2537 | −0.0009 |
| 10 | 047c0263… |
0.5622 | 0.5603 | −0.0019 |
| 11 | 049445b3… |
0.4437 | 0.4453 | +0.0016 |
| Average | 0.6149 | 0.6039 | −0.0110 |
Stage timings (mean of clips 2–11, ms; clip 1 excluded as warm-up)
| Stage | ONNX-TRT | Torch-TRT | Δ | Δ% |
|---|---|---|---|---|
| ViT | 364.0 | 365.4 | +1.4 | +0.4% |
| LLM Prefill | 890.7 | 875.0 | −15.7 | −1.8% |
| LLM Generation | 146.5 | 188.7 | +42.2 | +28.8% |
| Diffusor | 169.9 | 173.5 | +3.6 | +2.1% |
| E2E | 1571.9 | 1602.6 | +30.7 | +2.0% |
Engine sizes
| Engine | ONNX-TRT (MiB) | Torch-TRT (MiB) | Δ |
|---|---|---|---|
| LLM | 14484 | 14495 | +11 |
| Visual | 1106 | 1114 | +8 |
| Action | 4357 | 4380 | +23 |
Shared execution context (per-runner workspace, bytes)
| Runner | ONNX-TRT | Torch-TRT | Ratio |
|---|---|---|---|
| LLM | 2,776,893,952 | 2,776,893,952 | 1.0× |
| Vision | 4,074,504,192 | 4,074,505,216 | 1.0× |
| Action | 265,029,632 | 2,025,095,168 | ~7.6× |
Peak shared exec context is bounded by the vision runner (~4.07 GB) in both cases, so peak GPU memory is unchanged. Only the action runner's reserved workspace balloons under Torch-TRT.
Verdict
- Accuracy: equivalent on average (Torch-TRT actually 1 cm better on this 11-clip subset). Per-clip differences are sub-cm except for clip 5, where Torch-TRT happens to be 11 cm closer.
- Throughput: Torch-TRT is ~2% slower end-to-end. The regression is concentrated in LLM decode (+29%); prefill is actually marginally faster.
- Memory: action-runner workspace ~7.6× larger under Torch-TRT (peak GPU memory unchanged because vision dominates).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported Alpamayo-10B ONNX-TensorRT versus Torch-TensorRT benchmark, focusing on the LLM generation and action-runner workspace measurements. Trace the benchmark entry points and runtime configuration to identify the cause of the decode slowdown and workspace increase; done means a measured comparison shows the regression is addressed without changing the reported accuracy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100