rust-lang / rust-lang/rust

function definition subtyping can impact behavior

Open
#148,821 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A-coercions A-higher-ranked A-variance C-bug T-types
Dominant language
Rust
Stars
119k
Forks
16.2k
PR merge metrics
PR metrics pending

Description

The following code segfaults. I believe it is reasonable for users to expect this to be sound.

trait Trait {
    type Assoc: std::fmt::Display;
}
impl Trait for fn(&()) {
    type Assoc = Box<String>;
}
#[expect(coherence_leak_check)]
impl Trait for fn(&'static ()) {
    type Assoc = usize;
}

/// SAFETY: `value` must be a type erased object of type `T::Assoc`
unsafe fn interpret_value_as_assoc<T: Trait>(_: T, value: *mut ()) {
    // SAFETY: Guaranteed by safety requirement of this function
    let value: T::Assoc = unsafe { std::mem::transmute_copy(&value) };
    println!("{}", std::any::type_name::<T::Assoc>());
    println!("{value}");
}

fn call_me(x: Result<Box<String>, usize>) {
    let (func, value) = match x {
        Ok(val) => (
            interpret_value_as_assoc::<fn(&())>,
            Box::into_raw(val) as *mut (),
        ),
        Err(val) => (
            interpret_value_as_assoc::<fn(&'static ())>,
            val as *mut ()
        ),
    };

    // SAFETY: We use the correct function for either `Box<String>`
    // or `usize`.
    unsafe {
        func(|&()| (), value);
    }
}

fn main() {
    call_me(Ok(Box::new(String::from("hello there"))));
    call_me(Err(0));
}

interpret_value_as_assoc::<fn(&())> and interpret_value_as_assoc::<fn(&'static())> are subtypes of each other, even though they have fundamentally different behavior. The fact that the signatures of function definitions are subtypes is quite irrelevant for whether the actual behavior of that function is closely related.

cc @rust-lang/types

I am currently experimenting with @spastorino to remove subtyping which impacts behavior by replacing it with coercions. Currently subtyping can impact behavior either via TypeId or via trait selection. It would be great if that works out. Otherwise I would like to try to change function definitions to have invariant arguments

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the supplied Rust reproducer and investigate function-definition subtyping, trait selection, and the coherence_leak_check path. Compare the proposed coercion-based approach with invariant function arguments; done means the reproducer no longer permits behavior-changing subtype selection or a segfault.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.