[QNX] `Backtrace::{capture,force_capture}` triggers OOM on ARM64 when called from a thread other than `main`

The following program triggers an OOM condition when executed on a aarch64-unknown-nto-qnx710 target. The x86_64-pc-nto-qnx710 target is not affected by the issue. Other QNX targets have not been checked.

use std::backtrace::Backtrace;

// does OOM
fn main() {
    let handle = std::thread::spawn(|| {
        let trace = Backtrace::force_capture();
    });
    handle.join().unwrap();
}

$ ./gh130784
memory allocation of 3758096384 bytes failed

Calling the Backtrace::force_capture function from the main function / thread does not result in an OOM condition.

// no OOM
fn main() {
    let trace = Backtrace::force_capture();
    println!("{trace}");
}

$ ./gh130784-workaround
(..)
   2: std::backtrace::Backtrace::create
             at ./rustc/1.83.0/library/std/src/backtrace.rs:331:13
   3: error_with_backtrace_outputs_correctly::main
   4: std::sys::backtrace::__rust_begin_short_backtrace
   5: std::rt::lang_start::{{closure}}
(..)

The issue also affects the Backtrace::capture API when the RUST_BACKTRACE env var is set:

use std::backtrace::Backtrace;

// does OOM
fn main() {
    std::env::set_var("RUST_BACKTRACE", "1");
    let handle = std::thread::spawn(|| {
        let trace = Backtrace::capture();
    });
    handle.join().unwrap();
}

It's also worth noting that the built-in backtrace support (RUST_BACKTRACE=1) works fine on aarch64-unknown-nto-qnx710 since the /tests/ui/backtrace tests pass. Also, backtraces from panicking threads are printed as expected.

// no OOM
fn main() {
    std::env::set_var("RUST_BACKTRACE", "1");
    let handle = std::thread::spawn(|| {
        foo();
    });
    assert!(handle.join().is_err());
}

#[inline(never)]
fn foo() {
    panic!("backtrace works: {:x}", foo as fn() as usize);
}

$ ./rust-backtrace-works
thread '<unnamed>' panicked at <some-path>.rs:28:5:
backtrace works: 20223d58a8
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/1.83.0/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at ./rustc/1.83.0/library/core/src/panicking.rs:74:14
   2: <crate_name>::foo
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Changelog

• (2024-09-24 19:50:38+02:00) note that Backtrace::capture is also affected. note that the built-in RUST_BACKTRACE functionality is not affected when other threads panics.
• (2024-09-24 19:27:45+02:00) minimized repro example to a single call to Backtrace::force_capture. before it was replicating the std::error::tests::error_with_backtrace_outputs_correctly_with_one_source function which uses unstable libstd API.

cc @flba-eb @gh-tr @jonathanpallant (QNX 7.1 maintainers)
FYI @nyurik this may be relevant to the aarch64-unknown-nto-qnx700 target

Start with library/std/src/backtrace.rs around the Backtrace::create call shown in the report, then inspect the /tests/ui/backtrace tests. Reproduce the threaded force_capture and capture cases on aarch64-unknown-nto-qnx710 and compare them with main-thread and panic backtraces. Done means these APIs no longer trigger an OOM on the affected target, with coverage for the regression.