Report allocation errors through the panic handler

Proposal

Problem statement

We currently use a completely separate mechanism to handle allocation errors (OOM) than panics, despite both of these being very similar. However both are very similar: they involve handling runtime failures that are generally unrecoverable. It would be better to unify them under a single mechanism which is already stable: panic handlers (no_std) and panic hooks (std).

Motivation, use-cases

This change has already been done for no_std in https://github.com/rust-lang/rust/issues/66741: it was noticed that the vast majority of users of #[alloc_error_handler] were just calling panic anyways. This avoided the need to stabilize the alloc_error_handler attribute by defaulting to a panic if one was not specified.

The OOM hook API has also remained unstable for a very long time with no clear path to stabilization. The best way forward would be to unify it with the panic hook API just like it has been done in no_std.

Some use cases (firefox) still need access to the size of the failed allocation, so this is provided as a custom panic payload.

Solution sketches

// alloc::alloc

pub struct AllocErrorPanicPayload { .. }

impl AllocErrorPanicPayload {
    pub fn layout(&self) -> Layout;
}

This ACP proposes to make the following changes:

• Introduce the AllocErrorPanicPayload type which holds the Layout of a failed allocation.
• Make handle_alloc_error call the normal panic handler with a special payload of AllocErrorPanicPayload.
  • This can be observed by panic hooks with std or by the panic handler with no_std.
  • Unless -Zoom=panic is used, this is a non-unwinding panic: if the panic hook returns then the process is aborted. This maintains the current behavior of aborting on OOM by default.
• Remove the unstable alloc_error_hook std API (https://github.com/rust-lang/rust/issues/51245). Its job is now handled by panic hooks.
• Remove the unstable #[alloc_error_handler] attribute (https://github.com/rust-lang/rust/issues/51540) for no_std programs. Its job is now handled by #[panic_handler].

In the standard library, no additional allocations are made as part of the OOM handling process, except in 1 specific situation:

• With -Zoom=panic, if the panic hook returns then an allocation is necessary to box the payload and allocate an exception object. However it's not a big deal if this, fails, the process will simply abort immediately in that case.

Links and related work

Tracking issues that will be closed by this change:

• https://github.com/rust-lang/rust/issues/51245 alloc_error_hook
• https://github.com/rust-lang/rust/issues/51540 alloc_error_handler
  • Discussion about removing alloc_error_handler

What happens now?

This issue is part of the libs-api team API change proposal process. Once this issue is filed the libs-api team will review open proposals in its weekly meeting. You should receive feedback within a week or two.

Start with alloc::alloc::handle_alloc_error and the existing std panic hook and no_std panic handler APIs, then review tracking issues 51245 and 51540. Done means introducing AllocErrorPanicPayload with layout access, routing allocation failures through panic handling, and removing the proposed unstable allocation-error APIs while preserving abort behavior.