`fn(...) -> Option<T>`, where T is returned by invisible reference, could use null pointer for None
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 119k
- Forks
- 16.1k
- PR merge metrics
- PR metrics pending
Description
Consider a function that returns Option<T> where T is too large to be returned in registers, such as
const N: usize = 8;
pub fn fill_v(val: u64) -> Option<[u32; N]> {
let v = u32::try_from(val).ok()?;
Some(core::array::from_fn(|_| v))
}
Because [u32;8] doesn't fit in the ABI's return value registers, instead it's returned by "invisible reference"; at the assembly level, caller provides fill_v with two arguments, val and also a pointer to space for the return value. Right now, when you return Option<[u32;8]>, the Option discriminator takes up part of that space:
playground::fill_v: # @playground::fill_v
# %bb.0:
movq %rdi, %rax
xorl %ecx, %ecx
movq %rsi, %rdx
shrq $32, %rdx
jne .LBB0_2
# %bb.1:
movd %esi, %xmm0
pshufd $0, %xmm0, %xmm0 # xmm0 = xmm0[0,0,0,0]
movdqu %xmm0, 4(%rax)
movdqu %xmm0, 20(%rax)
movl $1, %ecx
.LBB0_2:
movl %ecx, (%rax)
retq
(The invisible reference is the first argument at the assembly level, in %rdi; the Rust-level first argument val is in %rsi.)
But the invisible reference argument is necessarily non-NULL, so we could apply the null pointer optimization for Option in this case, even though we can't do it in general for Option<[T;N]>, and generate code like this instead:
playground::fill_v: # @playground::fill_v
# %bb.0:
xorl %eax, %eax
movq %rsi, %rdx
shrq $32, %rdx
jne .LBB0_2
# %bb.1:
movd %esi, %xmm0
pshufd $0, %xmm0, %xmm0 # xmm0 = xmm0[0,0,0,0]
movdqu %xmm0, (%rdi)
movdqu %xmm0, 16(%rdi)
movq %rdi, %rax
.LBB0_2:
retq
This would save a word of stack space and would probably also be more efficient for the caller.
The example is an array, but the optimization is equally applicable to Option<LargeStruct>, etc.
Meta
rustc --version --verbose:
1.90.0-nightly (2025-07-21 9748d87dc70a9a6725c5)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the shown fill_v example with the recorded rustc nightly and compare its emitted assembly with the current and proposed forms. Trace the compiler entry points responsible for Option layout and invisible-reference returns; done means the optimization works for the demonstrated large return values without breaking other ABI cases, with regression coverage added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100