Next solver: recursion limit triggered on downstream... | rust-lang/rust

wgpu is a well known crate that currently requires increasing recursion limit with -Znext-solver=globally, in fact the latest release of it already intentionally includes #![recursion_limit = "256"].

However, turns out wgpu is infectious and causes any crate that directly or indirectly uses its Device type to trigger the same recursion limit warning (or error in older nightly).

I have reduced one of the crates in my workspace that uses wgpu here: https://github.com/nazar-pc/abundance/tree/reduced-wgpu-recursive-solver-recursion-limits

crates/farmer/ab-proof-of-space-gpu is a somewhat reduced library that uses wgpu:

use ab_core_primitives::pieces::Record;
use ab_core_primitives::sectors::SectorId;
use ab_farmer_components::plotting::RecordsEncoder;
use ab_farmer_components::sector::SectorContentsMap;
use rayon::{ThreadPool, ThreadPoolBuilder};
use std::sync::atomic::AtomicBool;

#[derive(Clone, Debug)]
pub struct Device;

impl Device {
    pub fn instantiate(&self) -> GpuRecordsEncoder {
        let thread_pool = ThreadPoolBuilder::new().build().unwrap();

        GpuRecordsEncoder {
            instances: Vec::new(),
            thread_pool,
        }
    }
}

#[derive(Debug)]
pub struct GpuRecordsEncoder {
    instances: Vec<GpuRecordsEncoderInstance>,
    thread_pool: ThreadPool,
}

impl RecordsEncoder for GpuRecordsEncoder {
    fn encode_records(
        &mut self,
        _sector_id: &SectorId,
        _records: &mut [Record],
        _abort_early: &AtomicBool,
    ) -> anyhow::Result<SectorContentsMap> {
        self.thread_pool.install(|| {
            let _ = &self.instances[0];
        });

        Ok(SectorContentsMap::new(0))
    }
}

#[derive(Debug)]
struct GpuRecordsEncoderInstance {
    _device: wgpu::Device,
}

Within this crate recursion is triggered here:

warning: overflow evaluating the requirement `{closure@crates/farmer/ab-proof-of-space-gpu/src/lib.rs:39:34: 39:36}: std::marker::Send`
  --> crates/farmer/ab-proof-of-space-gpu/src/lib.rs:39:26
   |
39 |         self.thread_pool.install(|| {
   |                          ^^^^^^^
   |
   = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_proof_of_space_gpu`)
   = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
   = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
   = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>
   = note: `#[warn(recursion_depth_exceeding_limit)]` (part of `#[warn(future_incompatible)]`) on by default

warning: overflow evaluating the requirement `{closure@crates/farmer/ab-proof-of-space-gpu/src/lib.rs:39:34: 39:36}: std::marker::Send`
  --> crates/farmer/ab-proof-of-space-gpu/src/lib.rs:39:9
   |
39 | /         self.thread_pool.install(|| {
40 | |             let _ = &self.instances[0];
41 | |         });
   | |__________^
   |
   = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_proof_of_space_gpu`)
   = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
   = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
   = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>

However, since wgpu::Device is a field of GpuRecordsEncoderInstance, which is stored inside public GpuRecordsEncoder, simply using GpuRecordsEncoder is enough to trigger recursion limits in further downstream crates, in this case crates/farmer/ab-farmer (both library and binary, triggered in both):

warning: overflow evaluating the requirement `tracing::instrument::Instrumented<{async block@crates/farmer/ab-farmer/src/plotter/gpu.rs:257:13: 257:23}>: std::marker::Send`
   --> crates/farmer/ab-farmer/src/plotter/gpu.rs:425:34
    |
425 |             AsyncJoinOnDrop::new(tokio::spawn(plotting_fut.in_current_span()), true);
    |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_farmer`)
    = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
    = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
    = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
    = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>
    = note: `#[warn(recursion_depth_exceeding_limit)]` (part of `#[warn(future_incompatible)]`) on by default

warning: overflow evaluating the requirement `std::pin::Pin<std::boxed::Box<{async block@crates/farmer/ab-farmer/src/plotter/gpu.rs:84:5: 84:10}>>: std::ops::CoerceUnsized<std::pin::Pin<std::boxed::Box<dyn futures::Future<Output = std::result::Result<bool, String>> + std::marker::Send>>>`
  --> crates/farmer/ab-farmer/src/plotter/gpu.rs:84:5
   |
84 |     async fn has_free_capacity(&self) -> Result<bool, String> {
   |     ^^^^^
   |
   = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_farmer`)
   = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
   = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
   = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>

warning: overflow evaluating the requirement `std::pin::Pin<std::boxed::Box<{async block@crates/farmer/ab-farmer/src/plotter/gpu.rs:88:5: 88:10}>>: std::ops::CoerceUnsized<std::pin::Pin<std::boxed::Box<dyn futures::Future<Output = ()> + std::marker::Send>>>`
  --> crates/farmer/ab-farmer/src/plotter/gpu.rs:88:5
   |
88 |     async fn plot_sector(
   |     ^^^^^
   |
   = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_farmer`)
   = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
   = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
   = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
   = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>

warning: overflow evaluating the requirement `std::pin::Pin<std::boxed::Box<{async block@crates/farmer/ab-farmer/src/plotter/gpu.rs:117:5: 117:10}>>: std::ops::CoerceUnsized<std::pin::Pin<std::boxed::Box<dyn futures::Future<Output = bool> + std::marker::Send>>>`
   --> crates/farmer/ab-farmer/src/plotter/gpu.rs:117:5
    |
117 |     async fn try_plot_sector(
    |     ^^^^^
    |
    = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_farmer`)
    = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
    = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
    = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
    = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>

warning: overflow evaluating the requirement `(parking_lot::lock_api::Mutex<parking_lot::RawMutex, std::vec::Vec<ab_proof_of_space_gpu::GpuRecordsEncoder>>, event_listener::Event): std::marker::Send`
  |
  = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_farmer`)
  = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
  = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
  = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
  = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>

warning: overflow evaluating the requirement `(parking_lot::lock_api::Mutex<parking_lot::RawMutex, std::vec::Vec<ab_proof_of_space_gpu::GpuRecordsEncoder>>, event_listener::Event): std::marker::Sync`
  |
  = help: consider increasing the recursion limit by adding a `#![recursion_limit = "256"]` attribute to your crate (`ab_farmer`)
  = help: or consider adding a manual `impl` of auto traits like `Send` for intermediate types, if auto traits are involved
  = note: this lint is attached to the whole crate and can't be disabled on a per-function basis
  = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
  = note: for more information, see issue #159228 <https://github.com/rust-lang/rust/issues/159228>

It is quite unexpected that downstream crates need to recursively also raise recursion limits in addition to the upstream dependency itself.

Next solver: recursion limit triggered on downstream packages

Description

Contributor guide

First steps

Research direction

Assessment