rust-lang / rust-lang/rust

Enum with non-primitive fields can be errorneously cast to primitive type

Open
#136,508 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-coercions C-bug T-compiler T-lang
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I tried this code:


#[repr(u8)]
pub enum FooBar {
    Foo(i32),
    Bar(String),
}

impl FooBar {
    fn discriminant(&self) -> u8 {
        unsafe { *(self as *const Self as *const u8) }
    }
}
fn main() {
    println!("{}", FooBar::Foo as u8);
    println!("{}", FooBar::Bar as u8);
    println!("{}", FooBar::Foo(42).discriminant());
    println!("{}", FooBar::Bar("baz".to_owned()).discriminant());
}

I expected to see this happen: The code should fail to compile, as casting to u8 for this enum is not allowed.

Instead, this happened: code compiles fine, and prints garbage. The garbage printed appears to be part of the address of the function that would instantiate the respective variant of the enum. As there are no warnings issued, this can be very nasty. Clippy does detect this and correctly warns.

Playground for easy reproduction on any compiler of choice.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=3e3b23e01f545caf4f4fc88ab45edd61

Note

  • Making any one field not contain any value will make this fail to compile (which is the expected result)
  • This can be especially nasty if you have an enum with no payload in any of the fields (which is safe to cast) and then add payload to all of the fields. The code will still compile, even though its meaning has changed completely.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Rust Playground and the FooBar casts in main, comparing them with the unsafe discriminant method. Trace how the compiler validates casts from repr(u8) enums with payloads. Done means the reproducer is rejected with an appropriate diagnostic and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.