numba / numba/llvmlite

Include compiler-rt in the JIT environment (with fix proposition)

Open
#495 9 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.3k
Forks
373
Avg merge
3d 2h
Merged PRs (30d)
10

Description

Hello everyone!

compiler-rt functions aren't included with llvmlite. This makes generated code that relies on them not to work.

# Reproducer

One example is the LLVM IR here: https://pastebin.com/SNgXV0Ph . It has been generated using clang 8 from the following code under Linux/x86_64: https://pastebin.com/k5tJ5NiT .

With the Python script here (https://pastebin.com/WcAy4BSv) and llvmlite 0.29.0, we get a segfault. Indeed, the `__udivti3` generated by clang (cf. https://godbolt.org/z/AbZ_92) is part of compiler-rt. As the LLVM dynamic loader can't find this function loaded within the process, it doesn't fix the `mov rXX, 0` instruction that is supposed to get the pointer to this function, and we end up with a call to 0.

This test case has been extracted from a unit case of the Miasm project (https://github.com/cea-sec/miasm/blob/master/test/arch/x86/unit/mn_div.py).

Note that this issue arises under Windows and Linux x86/64.

# Fix proposal 1

One way to fix this is to use `llvm::ExecutionEngine::addArchive`, with a pre-compiled compiler-rt/builtins static library. I made a quick'n'dirty POC here: https://github.com/aguinet/llvmlite/commit/89862c086b3120b30fa67a9aab0c8dc904d02952 . This indeed fixes the issue for Linux. I still have to try this under Windows.

# Fix proposal 2

One other fix that could work is to statically compile `compiler-rt` into the llvmlite .so/.dll binary. We can't use compiler-rt "as such" because we need to modify its CMake system a little bit to generate an archive with exported symbols. I tried an alternative of this under Linux by generating a .so file with the builtins symbols exported, and LD_PRELOAD it. This also fixes the bug. I don't if this would work on Windows though.

# Conclusion

Which fix would you prefer see implemented? Do you have other suggestions?

Thanks :)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked Python reproducer and LLVM IR, then inspect the proposed ExecutionEngine::addArchive approach and the llvmlite proof-of-concept commit. Compare the Linux and Windows behavior, including the Miasm mn_div.py test case; done means compiler-rt builtins resolve correctly without the reported segfault on both platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.