clang-nvlink-wrapper runs post-LTO ptxas serially; parallelize it or expose a jobs option
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
clang/tools/clang-nvlink-wrapper
In clang-nvlink-wrapper, the PTX files produced by the LTO backend are assembled one at a time. At
current main the loop in ClangNVLinkWrapper.cpp (around line 685) is:
```c++
for (StringRef LTOFile : LTOFiles) {
auto FileOrErr = runPTXAs(LTOFile, Args);
...
}
```
runPTXAs blocks in sys::ExecuteAndWait, so with --plugin-opt=lto-partitions=N the partitioning
parallelizes LLVM codegen and shrinks each ptxas input, but the assembly stage still runs N ptxas
processes back to back. Wall clock is the sum over partitions, not the max.
This matters for big RDC + -foffload-lto=full device images. Building PyTorch's libtorch_cuda.so
this way merges device bitcode from a few hundred TUs; we have measured a merged PTX around 920 MB
with ptxas peaking near 19 GB RSS, and at that size the assembly stage dominates the device link and
sits on one core long after the rest of the build has drained. nvlink's own -dlto path grew
--split-compile / --split-compile-extended for exactly this problem; the wrapper has no equivalent
surface. --plugin-opt=jobs= doesn't help here, it only sizes the ThinLTO backend pool, which a
full-LTO link never uses.
Two possible fixes, independent of each other:
1. Run the runPTXAs calls on a thread pool bounded by hardware_concurrency, keeping the output
order in Files stable. The invocations are independent of each other.
2. Forward ptxas's own --split-compile= through the -Xcuda-ptxas path, so even a single
monolithic PTX assembles multi-threaded. ptxas has accepted the flag since CUDA 12.x, and the
13.3 help text says it has "minimal (if any) impact on performance of the compiled binary".
To reproduce: any CUDA -fgpu-rdc -foffload-lto=full --offload-link link with
--plugin-opt=lto-partitions=16 shows 16 sequential ptxas invocations in ps or strace.
Contributor guide
Research direction
Start in clang/tools/clang-nvlink-wrapper/ClangNVLinkWrapper.cpp around the loop near line 685, and trace runPTXAs through sys::ExecuteAndWait. Reproduce with a CUDA -fgpu-rdc -foffload-lto=full link using --plugin-opt=lto-partitions=16, then verify that the chosen parallelization or jobs-option change removes serial ptxas execution while preserving stable output order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100