HWasan with external clang runtime (undefined symbol: __hwasan_tls)

Hi. I want to build with HWAsan and join c / rust / python code. I prepared example based on a public project https://github.com/pyca/cryptography

Steps to repro

• Clone cryptography
• Create config.toml
• Create Dockerfile
• Create hello.py
• Build it
• Run hello.py

Setup cryptography

git clone --recursive --branch=43.0.0 --depth=1 --single-branch git@github.com:pyca/cryptography.git
cd cryptography

config.toml

Create config.toml with content:

[build]
target="aarch64-unknown-linux-gnu"

rustflags = [
	"-g",
	"-Z", "sanitizer=hwaddress",
	"-Z", "external-clangrt",
	"-L", "/usr/lib/llvm-20/lib/clang/20/lib/linux/",
	"-l", "clang_rt.hwasan-aarch64",
	"-C", "link-arg=-fuse-ld=/usr/bin/ld.lld-20",
	"-C", "linker=/usr/bin/clang-20",
	"-C", "lto=no"
]

Dockerfile

Create Dockerfile with content:

FROM ubuntu:24.04 AS env
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update -y && \
    apt-get install -y \
        autoconf \
        cmake \
        curl \
        gnupg \
        libffi-dev \
        libssl-dev \
        lsb-release \
        ninja-build \
        patchelf \
        pkg-config \
        python3-dbg \
        python3-dev \
        python3-pip \
        python3-venv \
        software-properties-common \
        wget
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain nightly -y
ENV PATH=/root/.cargo/bin:$PATH
RUN wget https://apt.llvm.org/llvm.sh && \
    chmod +x llvm.sh && \
    ./llvm.sh 20 && \
    ln -s /usr/bin/lld-20 /usr/local/bin/lld && \
    ln -s /usr/bin/clang-20 /usr/local/bin/clang && \
    ln -s /usr/bin/clang++-20 /usr/local/bin/clang++ && \
    ln -s /usr/bin/clang-20 /usr/local/bin/cc && \
    ln -s /usr/bin/clang++-20 /usr/local/bin/c++ && \
    rm /usr/bin/ld && \
    ln -s /usr/lib/llvm-20/bin/ld.lld /usr/bin/ld
RUN python3 -m venv /venv
RUN echo "/usr/lib/llvm-20/lib/clang/20/lib/linux" > /etc/ld.so.conf.d/clang.conf && ldconfig
ENV CC=/usr/bin/clang-20
ENV CXX=/usr/bin/clang++-20
ENV CFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23"
ENV CCFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23"
ENV CXXFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c++23"
# ENV CPPFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0" | ?
ENV LDFLAGS="-fsanitize=hwaddress -shared-libsan"
ENV LDSHARED="/usr/bin/clang-20 -shared"
ENV RUSTFLAGS="-g -Zsanitizer=hwaddress -C linker=/usr/bin/clang-20 -C link-arg=-fuse-ld=/usr/bin/ld.lld-20 -C lto=no -Zexternal-clangrt -C target-feature=+tagged-globals"
ENV CARGO_BUILD_TARGET="aarch64-unknown-linux-gnu"
COPY config.toml /.cargo/

FROM env AS run
WORKDIR /src
COPY . .

Create hello.py (from examples) with content:

from cryptography.fernet import Fernet
key = Fernet.generate_key()
f = Fernet(key)

message = b"A really secret message. Not for prying eyes."
private_data = f.encrypt(message)
public_data = f.decrypt(token)

print(f"Message: {message}")
print(f"Private data: {private_data}")
print(f"Public data: {public_data}")

Build

$ docker build . -q
$ docker run --rm -it <image>

$ source /venv/bin/activate
$ cargo build
$ pip3 install setuptools
Collecting setuptools
  Downloading setuptools-73.0.1-py3-none-any.whl.metadata (6.6 kB)
Downloading setuptools-73.0.1-py3-none-any.whl (2.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 1.6 MB/s eta 0:00:00
Installing collected packages: setuptools
Successfully installed setuptools-73.0.1

$ pip3 install --no-binary ":all:" cffi --no-clean -vv
---------------- CUT ----------------
  ld: error: undefined symbol: __hwasan_init
  >>> referenced by _configtest.c
  >>>               _configtest.o:(hwasan.module_ctor)
  clang-20: error: linker command failed with exit code 1 (use -v to see invocation)

  Note: will not use '__sync_synchronize()' in the C code
  ***** The above error message can be safely ignored.
---------------- CUT ----------------
  ld: error: undefined symbol: __hwasan_init
  >>> referenced by _configtest.c
  >>>               _configtest.o:(hwasan.module_ctor)
  clang-20: error: linker command failed with exit code 1 (use -v to see invocation)
  Note: will not use '__sync_synchronize()' in the C code
  ***** The above error message can be safely ignored.
---------------- CUT ----------------
  ld: error: undefined symbol: __hwasan_init
  >>> referenced by _configtest.c
  >>>               _configtest.o:(hwasan.module_ctor)
  clang-20: error: linker command failed with exit code 1 (use -v to see invocation)
  Note: will not use '__sync_synchronize()' in the C code
  ***** The above error message can be safely ignored.
---------------- CUT ----------------
  building '_cffi_backend' extension
  creating build/temp.linux-aarch64-cpython-312
  creating build/temp.linux-aarch64-cpython-312/src
  creating build/temp.linux-aarch64-cpython-312/src/c
  /usr/bin/clang-20 -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -I/venv/include -I/usr/include/python3.12 -c src/c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-312/src/c/_cffi_backend.o
  src/c/_cffi_backend.c:4579:22: warning: 'Py_FileSystemDefaultEncoding' is deprecated [-Wdeprecated-declarations]
   4579 |                      Py_FileSystemDefaultEncoding, &filename_or_null, &flags))
        |                      ^
  /usr/include/python3.12/fileobject.h:22:1: note: 'Py_FileSystemDefaultEncoding' has been explicitly marked deprecated here
     22 | Py_DEPRECATED(3.12) PyAPI_DATA(const char *) Py_FileSystemDefaultEncoding;
        | ^
  /usr/include/python3.12/pyport.h:317:54: note: expanded from macro 'Py_DEPRECATED'
    317 | #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
        |                                                      ^
  In file included from src/c/_cffi_backend.c:8027:
  In file included from src/c/cffi1_module.c:20:
  src/c/call_python.c:211:5: warning: "no definition for read_barrier(), missing synchronization for multi-thread initialization in embedded mode" [-W#warnings]
    211 | #   warning "no definition for read_barrier(), missing synchronization for\
        |     ^
  2 warnings generated.
  /usr/bin/clang-20 -shared -fsanitize=hwaddress -shared-libsan -g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23 build/temp.linux-aarch64-cpython-312/src/c/_cffi_backend.o -L/usr/lib/aarch64-linux-gnu -lffi -o build/lib.linux-aarch64-cpython-312/_cffi_backend.cpython-312-aarch64-linux-gnu.so
  clang-20: warning: argument unused during compilation: '-mllvm -hwasan-globals=0' [-Wunused-command-line-argument]
---------------- CUT ----------------
  Created wheel for pycparser: filename=pycparser-2.22-py3-none-any.whl size=117552 sha256=d57055b6dddc795bb4eca6fc3754bb5ed521035680dd552d86560baed33ef091
  Stored in directory: /root/.cache/pip/wheels/36/53/17/c0ae2e096d359a9a8faf47fd7ded8f4c878af41a3c66cb5199
Successfully built cffi pycparser
Installing collected packages: pycparser, cffi
---------------- CUT ----------------
Successfully installed cffi-1.17.0 pycparser-2.22
Removed build tracker: '/tmp/pip-build-tracker-62wvhedt'

Run with pip isolation (by default):

$ pip3 install -e . --no-binary ":all:" --no-clean -vv
Using pip 24.0 from /venv/lib/python3.12/site-packages/pip (python 3.12)
---------------- CUT ----------------
= note: ld.lld-20: error: undefined symbol: __hwasan_tls
      >>> referenced by mod.rs:536 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ptr/mod.rs:536)
      >>>               /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(core::ptr::drop_in_place$LT$$LP$core..any..TypeId$C$alloc..boxed..Box$LT$dyn$u20$core..any..Any$u2b$core..marker..Sync$u2b$core..marker..Send$GT$$RP$$GT$::h0a3864e356d1b87e)
      >>> referenced by function.rs:250 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ops/function.rs:250)
      >>>               /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.00.rcgu.o:(core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::haffe049bd93073b5)
      >>> referenced by mod.rs:536 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ptr/mod.rs:536)
      >>>               /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(core::ptr::drop_in_place$LT$$LP$core..any..TypeId$C$alloc..boxed..Box$LT$dyn$u20$core..any..Any$u2b$core..marker..Sync$u2b$core..marker..Send$GT$$RP$$GT$::h0a3864e356d1b87e)
      >>> referenced 39817 more times

      ld.lld-20: error: undefined symbol: __hwasan_loadN
      >>> referenced by function.rs:250 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ops/function.rs:250)
      >>>               /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.00.rcgu.o:(core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::haffe049bd93073b5)
      >>> referenced by intrinsics.rs:3325 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/intrinsics.rs:3325)
      >>>               /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(_$LT$hashbrown..raw..RawTable$LT$T$C$A$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h059dda3c88ea07a7)
      >>> referenced by intrinsics.rs:3325 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/intrinsics.rs:3325)
      >>>               /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(_$LT$hashbrown..raw..RawTable$LT$T$C$A$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h059dda3c88ea07a7)
      >>> referenced 3842 more times
---------------- CUT ----------------
error: `cargo build --manifest-path Cargo.toml --message-format=json-render-diagnostics --target aarch64-unknown-linux-gnu --release -v --no-default-features --locked` failed with code 101

We can see that ENVs and /.cargo/config.toml are ignored and a lot of HWasan errors showed. I dont know how to fix it correctly (it is problem one; possibly, question to python community).

Run without pip isolation (but it is unreal to do in real world application):

$ pip3 install -e . --no-binary ":all:" --no-clean -vv --no-build-isolation
---------------- CUT ----------------
Successfully installed cryptography-43.0.0
Removed build tracker: '/tmp/pip-build-tracker-bvpypdpj' 

Run hello.py

$ python3 hello.py

Traceback (most recent call last):
  File "/src/hello.py", line 1, in <module>
    from cryptography.fernet import Fernet
  File "/src/src/cryptography/fernet.py", line 14, in <module>
    from cryptography.exceptions import InvalidSignature
  File "/src/src/cryptography/exceptions.py", line 9, in <module>
    from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: /src/src/cryptography/hazmat/bindings/_rust.abi3.so: undefined symbol: __hwasan_tls

We can see that ENVs and /.cargo/config.toml are not ignored, but __hwasan_tls (it is problem #2).

Additional information

$ rustc --version --verbose

rustc 1.82.0-nightly (eff09483c 2024-08-22)
binary: rustc
commit-hash: eff09483c67e6fc96c8098ba46dce476162754c5
commit-date: 2024-08-22
host: aarch64-unknown-linux-gnu
release: 1.82.0-nightly
LLVM version: 19.1.0

$ clang --version
Ubuntu clang version 20.0.0 (++20240821083450+84fa7b438e1f-1~exp1~20240821203619.364)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin

Few things which helps me it other cases:

• disable link-time-optimisation (otherwise link fails with R_AARCH64_ADR_PREL_PG_HI21 out of range)

• use only lld as linker (otherwise link fails R_AARCH64_ADR_PREL_PG_HI21 out of range)

• use shared (otherwise ASan conflicts or link fails)

• use external clang runtime (otherwise ASan conflicts)

• do not use gcc (sometimes it timeouts)

• pass target-feature=+tagged-globals (otherwise R_AARCH64_ADR_PREL_PG_HI21 out of range)

• pass -mllvm -hwasan-globals=0 (otherwise FP crashes)

• use global cargo config with environment variables (in case env-only, args do not passed to cargo/rustc) (pip isolation)

Reproduce the external HWASan setup using config.toml, Dockerfile, and hello.py, then compare cargo build with the pip isolated and non-isolated paths. Start with the reported linker errors for __hwasan_tls, __hwasan_init, and __hwasan_loadN. Done means the documented Rust/Python/C build and hello.py run without undefined HWASan symbols.