llvm / llvm/llvm-project

>4GB offsets in MCDataFragment in llvm-21 (llvm-dwp OOM)

Open
#168,923 4 comments 0 reactions 2 assignees Claimed by @tstellar View on GitHub
llvm:mc
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

We are having trouble with `llvm-dwp` running out of memory in llvm-21 for big builds resulting in giant dwp files where some sections are > 4GB in size. This makes creating a reproducer a bit tricky, but I'll describe what we are seeing:

It looks like recent MC layer rewrites, like https://github.com/llvm/llvm-project/commit/9beb467d9213fe2becdb0f1469d99ca058189ac7 introduce fields like `uint32_t ContentStart` to `MCEncodedFragment`. And my understanding is that in our case llvm-dwp ends up creating a single fragment bigger than 4GB after repeatet `emitBytes` calls here: https://github.com/llvm/llvm-project/blob/main/llvm/lib/DWP/DWP.cpp#L860
(The OOM situation seems to be accidental by `ContentEnd` overflowing causing `MCEncodedFragment::getContentsForAppending` to wrongly grow the contents vector to arbitrary size).

The code seems to have changed lately but I am still seeing various 32bit offsets in the fragment code. Should we increase those fields to 64bits or better find a way to not write everything into a single fragment in this use case?

CC @MaskRay

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.