Missed optimizations: Err returns should be out of line, loop not recognized as iterating at least once

As of today (2024-11-29), neither the release nor the nightly compiler
does an ideal job of optimizing this string validation function.

pub enum IdError {
    IdEmpty,
    IdTooLong,
    IdNotAscii,
}

#[inline(never)]
pub fn check_id(value: &str, limit: usize) -> Result<(), IdError> {
    if value.is_empty() {
        return Err(IdError::IdEmpty);
    }
    if value.len() > limit - 1 {
        return Err(IdError::IdTooLong);
    }
    if value.as_bytes().iter().any(|c| !(1u8..=127u8).contains(c)) {
        return Err(IdError::IdNotAscii)
    }
    Ok(())
}

There are two missed optimizations. For code like this, the compiler
would ideally recognize that conditional Err returns are unlikely,
and make the fall-through path be the one that (eventually) returns
Ok. Nightly gets this right for the first two checks but not the
third. Second, the compiler does not notice that, if the first two
checks succeed, then the loop in the third check must iterate at least
once. In addition to the redundant test, this might be inhibiting
vectorization of the loop.

The dec/cmp/jb sequence used for the second test can also be
micro-optimized to cmp/jbe but that’s extremely minor.

playground link

playground::check_id:
    testq   %rsi, %rsi
    je      .LBB0_1

    decq    %rdx
    movb    $1, %al      ; 1 = Err(IdTooLong)
    cmpq    %rsi, %rdx
    jb      .LBB0_8

    xorl    %eax, %eax
.LBB0_4:
    cmpq    %rax, %rsi
    je      .LBB0_5

    cmpb    $0, (%rdi,%rax)
    leaq    1(%rax), %rax
    jg  .LBB0_4

    movb    $2, %al      ; 2 = Err(IdNotAscii)
.LBB0_8:
    retq

.LBB0_1:
    xorl    %eax, %eax   ; 0 = Err(IdEmpty)
    retq

.LBB0_5:
    movb    $3, %al      ; 3 = Ok(())
    retq

playground::check_id:
    xor     %eax, %eax   ; 0 = Err(IdEmpty); recycled as index register
    test    %rsi, %rsi
    je      .L3
    cmp     %rsi, %rdx
    jbe     .L2

.L1:
    cmpb    $0, (%rdi,%rax)
    jng     .L4
    inc     %rax
    cmp     %rax, %rsi
    jb      .L1
    mov     $3, %al      ; 3 = Ok(())
    ret

.L2:
    mov     $1, %al      ; 1 = Err(IdTooLong)
.L3:
    ret
.L4:
    mov     $2, %al      ; 2 = Err(IdNotAscii)
    ret

Reproduce the issue with the linked Rust Playground on the stated nightly and release configurations, then inspect the generated assembly for check_id. Compare the control flow with the hand-optimized assembly in the report; done means the compiler removes the unnecessary loop-entry check and produces the intended fall-through error/Ok paths without regressing the existing checks.