rust-lang / rust-lang/rust

rustc and clang differ on noundef for hidden sret pointer parameters causing MSAN failures

Open
#162,902 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-ABI A-LLVM A-sanitizers C-bug
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

Running the below code in msan (-Zsanitizer=memory) fails MSAN. A simple Cargo-based repro case can be found at https://github.com/th0br0/bug-rustc-cc-interop-msan.

Failure:

==3140423==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 ...

companion.cc:

struct Large {
  uint64_t a[3];
};

extern "C" Large cpp_return_large(uint64_t x) {
  return Large{x, 0, 0};
}

main.rs

#[allow(dead_code)]
pub enum Padded {
    Small([u8; 1]),
    Large([u32; 1]),
}

#[repr(C)]
pub struct Large(pub [u64; 3]);
unsafe extern "C" {
    pub fn cpp_return_large(x: u64) -> Large;
}

#[inline(never)]
pub fn pass_padded(_: Padded) {}

fn main() {
    pass_padded(Padded::Small([1]));
    let large = unsafe { cpp_return_large(42) };
}

Automated root causing found that in rustc
https://github.com/rust-lang/rust/blob/main/compiler/rustc_target/src/callconv/mod.rs#L419 and https://github.com/rust-lang/rust/blob/main/compiler/rustc_codegen_llvm/src/abi.rs#L108 emit noundef but clang does not https://github.com/llvm/llvm-project/blob/main/clang/lib/CodeGen/CGCall.cpp#L3085 this then leads to this LLM-generated step-by-step:

  1. Step A: Rust calls a function by value with an 8-byte enum containing padding bytes (e.g. enum Padded { Small([u8; 1]), Large([u32; 1]) }, passed via PassMode::Cast(i64) without noundef), which stores non-zero padding shadow (0xffffff00_00000000) into __msan_param_tls[0].
  2. Step B: Rust calls a Clang-compiled extern "C" function that returns a > 16-byte struct via sret (Arg#0, %rdi). Because rustc marks the sret argument noundef, LLVM's MemorySanitizerPass::visitCallBase skips writing 0 to __msan_param_tls[0].
  3. Step C: Because Clang omits noundef on the sret parameter, MemorySanitizerPass in the C/C++ callee loads the stale shadow from __msan_param_tls[0] as the shadow for the sret pointer (%rdi) and traps when storing fields into *sret.

This supposedly can be fixed in either clang by emitting noundef or rust omitting noundef. I don't know which would be preferable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Cargo repro and run it under MSAN to confirm the failure. Read rustc_target/src/callconv/mod.rs around line 419 and rustc_codegen_llvm/src/abi.rs around line 108, then compare the noundef handling with Clang's CGCall.cpp around line 3085. Done means establishing the compatible ABI behavior and demonstrating that the repro no longer reports the stale-shadow MSAN failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.