llvm / llvm/llvm-project

clang-nvlink-wrapper runs post-LTO ptxas serially; parallelize it or expose a jobs option

Open
#216,890 0 comments 1 reaction 0 assignees View on GitHub
clang-tools-extra
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

clang/tools/clang-nvlink-wrapper

In clang-nvlink-wrapper, the PTX files produced by the LTO backend are assembled one at a time. At
current main the loop in ClangNVLinkWrapper.cpp (around line 685) is:

```c++
for (StringRef LTOFile : LTOFiles) {
auto FileOrErr = runPTXAs(LTOFile, Args);
...
}
```

runPTXAs blocks in sys::ExecuteAndWait, so with --plugin-opt=lto-partitions=N the partitioning
parallelizes LLVM codegen and shrinks each ptxas input, but the assembly stage still runs N ptxas
processes back to back. Wall clock is the sum over partitions, not the max.

This matters for big RDC + -foffload-lto=full device images. Building PyTorch's libtorch_cuda.so
this way merges device bitcode from a few hundred TUs; we have measured a merged PTX around 920 MB
with ptxas peaking near 19 GB RSS, and at that size the assembly stage dominates the device link and
sits on one core long after the rest of the build has drained. nvlink's own -dlto path grew
--split-compile / --split-compile-extended for exactly this problem; the wrapper has no equivalent
surface. --plugin-opt=jobs= doesn't help here, it only sizes the ThinLTO backend pool, which a
full-LTO link never uses.

Two possible fixes, independent of each other:

1. Run the runPTXAs calls on a thread pool bounded by hardware_concurrency, keeping the output
order in Files stable. The invocations are independent of each other.
2. Forward ptxas's own --split-compile= through the -Xcuda-ptxas path, so even a single
monolithic PTX assembles multi-threaded. ptxas has accepted the flag since CUDA 12.x, and the
13.3 help text says it has "minimal (if any) impact on performance of the compiled binary".

To reproduce: any CUDA -fgpu-rdc -foffload-lto=full --offload-link link with
--plugin-opt=lto-partitions=16 shows 16 sequential ptxas invocations in ps or strace.

Contributor guide

Open the contributing guide

Research direction

Start in clang/tools/clang-nvlink-wrapper/ClangNVLinkWrapper.cpp around the loop near line 685, and trace runPTXAs through sys::ExecuteAndWait. Reproduce with a CUDA -fgpu-rdc -foffload-lto=full link using --plugin-opt=lto-partitions=16, then verify that the chosen parallelization or jobs-option change removes serial ptxas execution while preserving stable output order.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.