[Reproducible Builds] lwk non-determinism
- Dominant language
- Rust
- Stars
- 111
- Forks
- 60
- Avg merge
- 1h 32m
- Merged PRs (30d)
- 1
Description
While verifying the reproducibility of Blockstream Green desktop (green_qt 3.4.0 AppImage) for [WalletScrutiny](https://walletscrutiny.com), we found that `liblwk.so` does not build deterministically: two consecutive `cargo build --locked --release -p lwk_bindings` runs from the **same source tree, same toolchain, same container, and same target path** produce binaries differing by **2,511,912 bytes** (of ~32 MB unstripped). This prevents deterministic rebuild verification of the bundled `usr/lib/liblwk.so` using the published recipe.
## Reproducer
Environment (green_qt CI-equivalent container):
- Base: **Ubuntu 22.04.5 LTS (jammy)**, x86_64
- rustc **1.85.0** (`4d91de4e48198da2e33413efdcd9cd2cc0c46688`, 2025-02-17), host `x86_64-unknown-linux-gnu`, **LLVM 19.1.7**
- cargo 1.85.0 (`d73d2caf9`, 2024-12-31)
- cc: gcc (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0; linker: GNU ld (GNU Binutils for Ubuntu) **2.38**
```bash
git clone https://github.com/Blockstream/lwk /lwk-test/src
cd /lwk-test/src
git checkout bdbf0904466c51985f262d8ad2d5436951d80a3e # wasm_0.17.1-87-gbdbf0904 (pinned by green_qt 3.4.0)
git submodule sync && git submodule update --init --recursive
CARGO_TARGET_DIR=/lwk-test/t5 cargo build --locked --release -p lwk_bindings
cp /lwk-test/t5/release/liblwk.so /lwk-test/t5-first.so
cargo clean --target-dir /lwk-test/t5
CARGO_TARGET_DIR=/lwk-test/t5 cargo build --locked --release -p lwk_bindings
cmp /lwk-test/t5-first.so /lwk-test/t5/release/liblwk.so # differs
```
Note the second build uses the **identical** target path, eliminating `CARGO_TARGET_DIR`/path-embedding as a cause (we tested cross-path pairs too — same result).
## Evidence
- GNU Build IDs: `3ce1ba644a3045a52414db4a028975bd6902cfab` vs `e946e9d012caf4f58d3257fdb69ab161c28c6c24` (GNU ld `--build-id` sha1 — a content hash, so the divergence is real content, not just the note section).
- 2,511,912 differing bytes in 49 clusters, mapping (`readelf -SW` + `cmp -l`) to: `.dynsym`, `.text`, `.rodata`, `.eh_frame_hdr`, `.eh_frame`, `.gcc_except_table`, `.symtab`, `.strtab` — i.e. symbol tables, code, and unwind data shift together.
- The string-level diff is the tell: **94 of 95 differing strings are the same Rust symbols with different `.llvm.` suffixes**, e.g.:
- build 1: `_ZN42_..Debug$GT$3fmt17hc04d7db5617ccf91E.llvm.6939166226008123532`
- build 2: same symbols with `.llvm.7473467968691498770` / `.llvm.15652946576325426135`
- `.llvm.` suffixes are the promoted-local-symbol module hashes added by **ThinLTO**. `lwk_bindings`' release profile sets no explicit `lto` key, so cargo defaults to **thin-local LTO**.
- **LTO-off control confirms the mechanism:** with `CARGO_PROFILE_RELEASE_LTO=off` (same-path pair), the diff collapses from 2,511,912 bytes to **3,361 bytes**, the `.llvm.` suffixes disappear from the binary entirely, and the string-level diff drops to zero. ThinLTO accounts for 99.87% of the divergence; a small residual (a few `.dynsym`/`.symtab` entries and isolated code/data spots) remains.
- `codegen-units = 1` alone (default LTO, same-path pair): also still differs — no `.llvm.` suffixes are observed in these binaries either, yet the two builds differ **in size** by 24 bytes (25,122,696 vs 25,122,720), and the only string-level diff is again `GCC_except_table` label renumbering (one build has an extra label).
- `LTO=off` + `codegen-units=1` combined: still differs by ~6 KB (21 clusters). The string-level diff here is suggestive: the only differing strings are **`GCC_except_table` local labels with shifted numbers** (e.g. `GCC_except_table2142` vs `GCC_except_table2145`). This is consistent with unstable LLVM/backend symbol numbering or emission order; the residual cause remains unidentified and is **independent of both the LTO mode and the codegen-unit count**.
## Why it matters
green_qt bundles `liblwk.so` in the official AppImage. Everything else in the 3.4.0 AppImage payload that differed in our rebuild traces to timestamps or packaging-tool versions (separate report/issue), but liblwk's diff is build-nondeterminism: repeated controlled builds produced different binaries, so the recipe cannot currently provide deterministic verification. A deterministic liblwk would make the Green desktop AppImage realistically verifiable end-to-end.
## Suggested next steps
1. As a **diagnostic mitigation** (not a complete fix — see point 3): `lto = "off"` in the release profile removes 99.87% of the divergence (the ThinLTO module-hash churn). Note the [cargo profile docs](https://doc.rust-lang.org/cargo/reference/profiles.html#lto): `lto = false` still enables thin-local LTO; only `lto = "off"` disables LTO entirely — which is why the default release profile (no `lto` key) exhibits the `.llvm.` churn.
2. Investigate the ThinLTO module-hash instability under rustc 1.85.0 / LLVM 19.1.7 (we can provide all binary pairs and full logs).
3. A small residual divergence (visible as `GCC_except_table` renumbering, consistent with unstable symbol numbering/emission order — in one configuration it even changes the binary size by 24 bytes) survives **every combination we tested**: `lto=off` alone, `codegen-units=1` alone, and both together. Its cause remains unidentified — possible directions: build parallelism scheduling, a build script or proc-macro with nondeterministic iteration order, or a vendored C/C++ dependency.
We're happy to share all six `.so` files (both builds of each test pair), full build logs, and the diff analysis.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.