Possibly hitting open file limits on Linux with lld and large archives + many CPUs
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
I'm doing an LTO build of qemu on a machine with 64 cores. There is an archive in qemu that has 636 objects in it and when there are several lld jobs running in parallel all trying to link the archive into separate binaries, it's resulting in at least 600,000 open files which is
more than the limit reported by ulimit. The build is failing with the error: `clang: error: unable to execute command: posix_spawn failed: Resource temporarily unavailabl`, but I think the number of open files may be the root cause of this failure.
I've attached the output of lsof showing all the files open by lld.
It looks like each object in the archive is being opened in every lld thread, so that would mean 64 * 636 = 40,704 open files in each lld process.
[lsof.qemu.lld.log.xz.txt](https://github.com/user-attachments/files/30380914/lsof.qemu.lld.log.xz.txt)
Contributor guide
Research direction
No source file or test is named in the report. Start by reproducing the qemu LTO build with multiple lld jobs on Linux and inspect the attached lsof output, then trace the lld path that opens archive members; done means establishing whether this causes the resource failure and identifying the affected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100