Type inference not working well with ptr mutability

I tried this code:

fn main() {
    let mut tmp: usize = 3;
    let immut_addr = &tmp as *const _;
    let addr = immut_addr.cast_mut();
}

I expected to see this happen: the code compiles successfully (but with unused warnings)

Instead, this happened: The compiler fails to infer the type of addr because it doesn't recognize that the type it inferred for immut_addr is always the same as the one cast_mut produces

On stable the compiler suggests to add : *const _ to immut_addr as a type annotation, but after adding this, it errors the same way it did before.

Meta

rustc --version --verbose:
This happens both on stable (1.78) and on the latest nightly

rustc 1.80.0-nightly (1fffb2a35 2024-04-28)
binary: rustc
commit-hash: 1fffb2a355dfee65bf6aa5a9db35f33d3ee93aec
commit-date: 2024-04-28
host: x86_64-unknown-linux-gnu
release: 1.80.0-nightly
LLVM version: 18.1.4

error[E0699]: cannot call a method on a raw pointer with an unknown pointee type
 --> src/main.rs:6:27
  |
6 |     let addr = immut_addr.cast_mut();
  | 

Start by reproducing the snippet in src/main.rs with the reported stable and nightly rustc versions, then investigate how raw-pointer type inference handles cast_mut. Done means the example compiles without requiring an annotation and the behavior is covered by an appropriate compiler regression test.