rust-lang / rust-lang/rust

`fn(...) -> Option<T>`, where T is returned by invisible reference, could use null pointer for None

Open
#144,310 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

C-discussion
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

Consider a function that returns Option<T> where T is too large to be returned in registers, such as

const N: usize = 8;

pub fn fill_v(val: u64) -> Option<[u32; N]> {
    let v = u32::try_from(val).ok()?;
    Some(core::array::from_fn(|_| v))
}

Because [u32;8] doesn't fit in the ABI's return value registers, instead it's returned by "invisible reference"; at the assembly level, caller provides fill_v with two arguments, val and also a pointer to space for the return value. Right now, when you return Option<[u32;8]>, the Option discriminator takes up part of that space:

playground::fill_v: # @playground::fill_v
# %bb.0:
	movq	%rdi, %rax
	xorl	%ecx, %ecx
	movq	%rsi, %rdx
	shrq	$32, %rdx
	jne	.LBB0_2
# %bb.1:
	movd	%esi, %xmm0
	pshufd	$0, %xmm0, %xmm0                # xmm0 = xmm0[0,0,0,0]
	movdqu	%xmm0, 4(%rax)
	movdqu	%xmm0, 20(%rax)
	movl	$1, %ecx

.LBB0_2:
	movl	%ecx, (%rax)
	retq

(The invisible reference is the first argument at the assembly level, in %rdi; the Rust-level first argument val is in %rsi.)

But the invisible reference argument is necessarily non-NULL, so we could apply the null pointer optimization for Option in this case, even though we can't do it in general for Option<[T;N]>, and generate code like this instead:

playground::fill_v: # @playground::fill_v
# %bb.0:
	xorl	%eax, %eax
	movq	%rsi, %rdx
	shrq	$32, %rdx
	jne	.LBB0_2
# %bb.1:
	movd	%esi, %xmm0
	pshufd	$0, %xmm0, %xmm0                # xmm0 = xmm0[0,0,0,0]
	movdqu	%xmm0, (%rdi)
	movdqu	%xmm0, 16(%rdi)
	movq	%rdi, %rax
.LBB0_2:
	retq

This would save a word of stack space and would probably also be more efficient for the caller.

The example is an array, but the optimization is equally applicable to Option<LargeStruct>, etc.

Meta

rustc --version --verbose:

1.90.0-nightly (2025-07-21 9748d87dc70a9a6725c5)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the shown fill_v example with the recorded rustc nightly and compare its emitted assembly with the current and proposed forms. Trace the compiler entry points responsible for Option layout and invisible-reference returns; done means the optimization works for the demonstrated large return values without breaking other ABI cases, with regression coverage added.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.