rust-lang / rust-lang/rust

Missed optimization: returning a niche-optimized `Option<&T>` leaves a redundant null check after inlining

Open
#159,335 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

The following code:

use core:hint::black_box;

#[inline(always)]
fn foo(cond: bool, ptr: *const usize) -> Option<&'static usize> {
    if cond {
        Some(unsafe { &*ptr })
    } else {
        None
    }
}

#[unsafe(no_mangle)]
fn bar(cond: bool, ptr: *const usize) {
    if let Some(r) = foo(cond, ptr) {
        black_box(r);
    }
}

is compiled with -C opt-level=3 -Zinline-mir=no into the following assembly:

bar:
        push    rax
        xor     cl, 1
        test    rdx, rdx
        sete    al
        or      al, cl
        jne     .LBB0_2
        mov     qword ptr [rsp], rdx
        mov     rax, rsp
.LBB0_2:
        pop     rax
        ret

In the assembly, there are two checks merged in one, the first is the boolean condition, and the second one is whether ptr is null. The second check is redundant and should ideally not be compiled.

https://godbolt.org/z/esxPahrzE

My investigations so far

The issue doesn't happen when foo is inlined in the MIR, that's why MIR inlining is disabled. It can be reproduced without the compiler flag by adding a bit more code to foo, so it is not inlined in the MIR but by LLVM. That's what happened my original reproducing code.

Because totally disabling has some unwanted side effects, in my next godbolt link, I add 17 black_box(cond) at the beginning of foo to make it not MIR-inlinable: https://godbolt.org/z/Gxqhb3cqM

The problem seems to come from the niche optimization of Option<&T>. In fact, I've tried many things in godbolt, like replacing the return type by Option<Cell<&T>> or (MaybeUninit<T>, bool), both removing the niche optimization, and in these cases, the redundant check was not compiled.

Making LLVM understand that the pointer is not null also removes the redundant check. For example, replacing &*ptr by ptr.as_ref().unwrap_unchecked(), or wrapping it into a black_box work.

One strange thing is that adding an assert_unchecked(!ptr.is_null()) has no effect. However, if ptr becomes the field of a struct, assert_unchecked(!self.ptr.is_null()) correctly tag the self.ptr as !nonnull and eliminate the redundant check. My own code is in this case (the pointer is a field), so I use the assert_unchecked solution for now.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducer and the linked Godbolt examples, comparing output with MIR inlining disabled and enabled. Investigate how the niche optimization for Option<&T> is represented through LLVM inlining; done means the redundant null check is eliminated without requiring MIR inlining or caller-side workarounds.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.