rust-lang / rust-lang/rust

Layout optimization fails for enums with variants having values 0 and 255

Open
#136,444 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-layout C-bug C-optimization
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

When a C-like enum has two variants, one being set to 0 and the other being set to 255 the layout optimizations fail to apply even though they do apply if the values are different.

I tried this code:

struct Foo {
    _bar: Bar,
    _baz: [u8; 7],
}

enum Bar {
    _A = 0,
    _B = 255,
}

struct Error {
    _x: bool,
    _y: bool,
}

fn main() {
    println!("{}", core::mem::size_of::<Result<Foo, Error>>());
}

I expected to see this happen: 8 being printed because of layout optimization, since all values 1..=254 are invalid the optimizer can pick any of them to represent the Err variant and store the error data in the space that would be otherwise occupied by [u8; 7]. This already works if the values are different.

Instead, this happened: The output is 9, implying the optimization was not applied. If you change the value of _A or _B then the optimization is applied (prints 8).

Meta

rustc --version --verbose:

rustc 1.86.0-nightly (ae5de6c75 2025-01-29)
binary: rustc
commit-hash: ae5de6c759cd337ecdb2de4e94f47eaafb5d4606
commit-date: 2025-01-29
host: x86_64-unknown-linux-gnu
release: 1.86.0-nightly
LLVM version: 19.1.7

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by compiling the provided Rust reproducer and checking its reported Result<Foo, Error> size. Trace the enum layout optimization for variants with values 0 and 255; the issue is resolved when this case produces the optimized size of 8 without regressing cases with different values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.