Deadlock when capturing a backtrace from allocator during panic with test output capturing enabled

I tried this code (minimized from a larger testcase):

use std::alloc::{GlobalAlloc, Layout};
use std::cell::Cell;
use std::backtrace::Backtrace;
use std::thread_local;

thread_local! {
    static CAN_ALLOCATE: Cell<bool> = const { Cell::new(true) };
}

#[derive(Debug)]
pub struct NoAllocate(std::alloc::System);

unsafe impl GlobalAlloc for NoAllocate {
    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
        if !CAN_ALLOCATE.replace(true) {
            let _ =  Backtrace::force_capture();
        }
        self.0.alloc(layout)
    }

    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
        self.0.dealloc(ptr, layout);
    }
}

#[global_allocator]
static GLOBAL: NoAllocate = NoAllocate(std::alloc::System);

#[test]
#[should_panic]
fn main() {
    CAN_ALLOCATE.set(false);
    panic!();
}

Compiled with rustc --test main.rs.

I expected to see this happen: the test successfully panics and exits.

Instead, this happened: the test deadlocks.

Relevant section of backtrace:

...
#4  std::sync::mutex::Mutex::lock<()> () at std/src/sync/mutex.rs:317
#5  std::sys::backtrace::lock () at std/src/sys/backtrace.rs:18
#6  std::backtrace::Backtrace::create () at std/src/backtrace.rs:326
#7  0x0000555555585b25 in std::backtrace::Backtrace::force_capture () at std/src/backtrace.rs:312
#8  0x000055555556ab52 in <main::NoAllocate as core::alloc::global::GlobalAlloc>::alloc ()
#9  0x000055555556ac45 in __rust_alloc ()
...
#15 alloc::vec::Vec::reserve<u8, alloc::alloc::Global> () at alloc/src/vec/mod.rs:973
...
#22 0x00005555555879a3 in std::io::Write::write_fmt<alloc::vec::Vec<u8, alloc::alloc::Global>> () at std/src/io/mod.rs:1823
#23 0x000055555558aeb7 in std::panicking::default_hook::{closure#1} () at std/src/panicking.rs:256

This is specifically related to output capturing from the test runner, running the same code as a non-test binary or with --nocapture works perfectly.

Meta

This worked in 1.80.1 and nightly-2024-07-13, it started failing in 1.81.0 and nightly-2024-07-14.

The deadlock was introduced by https://github.com/rust-lang/rust/pull/127397 (cc @jyn514).

The lock is first taken when starting to print from the default panic hook:

https://github.com/rust-lang/rust/blob/26b2b8d162ff6cc399c7e8005230beea7eb12037/library/std/src/panicking.rs#L257

The first print to the output then happens:

https://github.com/rust-lang/rust/blob/26b2b8d162ff6cc399c7e8005230beea7eb12037/library/std/src/panicking.rs#L258

For captured output this requires then reallocating the Vec storing the capture, so it calls into the allocator, hitting the Backtrace::force_capture because this is the first allocation in the test. This attempts to re-entrantly acquire the lock a second time, deadlocking:

https://github.com/rust-lang/rust/blob/26b2b8d162ff6cc399c7e8005230beea7eb12037/library/std/src/backtrace.rs#L326

Reproduce the deadlock with rustc --test using the minimized allocator example and captured output. Read library/std/src/panicking.rs, library/std/src/backtrace.rs, and library/std/src/sys/backtrace.rs around the linked lines, then add regression coverage showing the test completes with output capture enabled without re-entrant locking.