rust-lang / rust-lang/rust

Inefficient code generation for u8 -> boolean conversion if bounds check already applied

Open
#121,673 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-LLVM C-bug C-optimization S-has-mcve T-compiler
Dominant language
Rust
Stars
119k
Forks
16.2k
PR merge metrics
PR metrics pending

Description

This occurs both on the latest stable and nightly with both -C opt-level=2 and -C opt-level=3, and looking back some versions this has been going on for quite a while, so this is not a regression AFAIK. Consider the following piece of code:

#[no_mangle]
pub fn test(x: &u8) -> bool {
    let load = *x;
    if load <= 1 {
        return load == 1;
    }

    // Just some nonsense to keep the above branch.
    something_dynamic()
}

#[inline(never)]
#[cold]
fn something_dynamic() -> bool {
    std::env::var("dynamic").is_ok()
}

I would expect that in a release build, the above code loads the u8, checks if it is a legal value for a bool, and if so return it, otherwise going to something_dynamic(). The motivation for this piece of code is something similar to a OnceLock<bool> in an incredibly hot piece of code. Ideally the hot path only consists of a single mov, cmp and jmp. Compiled we see the following:

test:
        movzx   eax, byte ptr [rdi]
        cmp     al, 2
        jae     example::something_dynamic
        cmp     al, 1
        sete    al
        ret

example::something_dynamic:
        <omitted>

It appears that Rust always emits code for turning an u8 into a bool (cmp al, 1 followed by sete al), regardless of whether this is necessary. In this case, since we checked that the u8 is <= 1, this is not necessary at all. The same problem occurs on ARM:

_test:
        ldrb    w8, [x0]
        cmp     w8, #2
        b.hs    LBB0_2
        cmp     w8, #1
        cset    w0, eq
        ret
       
LBB0_2:
        b       example::something_dynamic

The problem even persists if we try to use transmute, or pointer casts to bypass the problem. All the following variants still cause a superfluous value test:

    if load <= 1 {
        return load == 1;
        // return unsafe { *(&load as *const u8 as *const bool) };
        // return unsafe { std::mem::transmute(load) };
        // return unsafe { std::mem::transmute_copy(&load) };
    }

The weirdest thing is that this pessimization only occurs when there is a bounds check already applied. The following function avoids a test, directly loading the byte:

#[no_mangle]
unsafe fn maybe_load(x: &u8, should_load: bool) -> bool {
    if should_load {
        return unsafe { std::mem::transmute_copy(x) };
    }

    something_dynamic()
}
maybe_load:
        test    esi, esi
        je      example::something_dynamic
        movzx   eax, byte ptr [rdi]
        ret

However, if you try to implement test in terms of maybe_load, it gets inlined and the useless cmp, sete is back.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Rust reproducer and compare its optimized x86 and ARM output using the linked Compiler Explorer example. Trace the generated boolean conversion and bounds-check handling in the compiler codegen path. Done means the redundant boolean normalization is removed when the existing bounds check proves the value is 0 or 1, without changing the fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.