llvm / llvm/llvm-project

[DTLTO][COFF] clang-cl misreads the serialized remote compiler options: -O2 becomes -O3, -ffunction-sections/-fdata-sections/-floop-interchange are dropped

Open
#220,379 5 comments 0 reactions 0 assignees View on GitHub
lld:COFF LTO
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

I noticed our compiler arguments weren't getting through when using DTLTO. The AI writeup is below.

---

For a COFF DTLTO link, `buildCommonRemoteCompilerOptions()` serializes the backend compiler invocation as:

`clang-cl.exe -c --target=x86_64-pc-windows-msvc19.51.36256 -O2 -faddrsig -ffunction-sections -fdata-sections -floop-interchange -Wno-unused-command-line-argument`

When the remote compiler is clang-cl, two of these options are misinterpreted and three are dropped, so the distributed backends do not reproduce the in-process ThinLTO backend.

### 1. Three of the serialized flags are not accepted by clang-cl

Every COFF DTLTO backend job prints:

clang-cl: warning: unknown argument ignored in clang-cl: '-ffunction-sections' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-fdata-sections' [-Wunknown-argument]
clang-cl: warning: unknown argument ignored in clang-cl: '-floop-interchange' [-Wunknown-argument]

The options are dropped. `lld/COFF/LTO.cpp` sets `FunctionSections`, `DataSections` and (by default) `PTO.LoopInterchange` for the in-process backend, and the serializer faithfully emits them, but in GCC spelling. The backend objects therefore come back with a single `.text` and a single `.data` section per module, and `/OPT:REF` and `/OPT:ICF` cannot operate at function or data granularity inside them.

### 2. `-O2` means `-O3` to clang-cl

`-O` is emitted as `-O2` for lld's default level. clang-cl treats `-O2` as `/O2`, and `TranslateOptArg` in `clang/lib/Driver/ToolChains/MSVC.cpp` expands `/O2`, `/Ox` and `/Ot` to cc1 `-O3`. So every clang-cl backend runs the O3 pipeline with aggressive codegen, while the in-process backend runs O2 (`LTOBackend.cpp` maps `Conf.OptLevel` 0-3 directly). Verified with `clang-cl -### -c -x ir`: `-O2`, `/O2`, `/Ox` and `/Ot` all produce cc1 `-O3`; only `/clang:-O2` (or `-Xclang -O2`) produces `-O2`. There is no cl-style `/O` flag that yields `-O2`.

Note that there is nothing to translate for size configurations: as of LLVM 23 there are no Os/Oz pipelines (#191363, #209424), and size intent is carried by the `optsize`/`minsize` attributes already in the bitcode, which the backend honors at any level.

### Measurements

Large application (~3,000 modules, 480,230 text functions), Win64, same bitcode objects and link flags, DTLTO vs in-process ThinLTO, clang/lld 23.1.0, attributing the difference per option via `/thinlto-remote-compiler-arg:`:

| remote options appended | `.text` | text functions | vs in-process |
|---|---|---|---|
| none (serialized options only) | 525.0 MB | 540,115 | 60,161 extra functions, no function granularity for /OPT:REF and /OPT:ICF |
| `/Gy /Gw` | 462.9 MB | 480,230 | function set matches, code 5.1% larger |
| `/Gy /Gw /clang:-O2` | 440.4 MB | 480,230 | `.text`, `.data`, `.pdata` byte-identical; all functions identical in size |

In-process: `.text` 440.4 MB, 480,230 functions. The same identity holds for `-Os` and for PGO instrumented and optimized configurations without any further option, confirming that size and profile intent travel in the IR. `-floop-interchange` being dropped is real but negligible on this code: disabling the pass in-process changes 3 functions and 224 bytes.

With the level and sections fixed, the one remaining difference is section order under `/call-graph-profile-sort` for PGO builds, because the sort breaks density ties on the order in which edges are read and the two paths deliver backend objects in different orders; with `/call-graph-profile-sort:no` the images are byte-identical.

### Suggestions

- When the remote compiler runs in cl driver mode (by name or by an explicit option), emit `/clang:-O` for the level and `/Gy`, `/Gw`, `/clang:-floop-interchange` (or `/clang:`-prefixed GCC spellings throughout) for the code generation options. `--thinlto-remote-compiler-prepend-arg` shows the plumbing already exists for driver-specific arguments.
- Treat a rejected translation option as an error rather than hiding it behind `-Wno-unused-command-line-argument`; today the rejection only surfaces because clang-cl reports it under `-Wunknown-argument`, and a build system that suppresses that warning loses the signal entirely.

Observed with clang/lld 23.1.0. Related: #220371.

Contributor guide

Open the contributing guide

Research direction

Start by tracing option serialization in lld/COFF/LTO.cpp and the clang-cl translation behavior in clang/lib/Driver/ToolChains/MSVC.cpp. Use the documented DTLTO remote compiler invocation and --thinlto-remote-compiler-arg plumbing to verify that optimization, section, and loop-interchange options reach the backend with equivalent behavior to in-process ThinLTO.

Written by the indexing model from the issue text.

Assessment

Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.