bevyengine / bevyengine/bevy

Support asynchronous Picking backends

Open
#17,926 6 comments 0 reactions 0 assignees View on GitHub
A-Picking C-Feature S-Needs-Design
Dominant language
Rust
Stars
48.2k
Forks
4.8k
Avg merge
3d 22h
Merged PRs (30d)
161

Description

## What problem does this solve or what need does it fill?

We have the following constraints:
- We display very large 3D models stored in a remote server.
- We can't store the 3D mesh in RAM so we set `RenderAssetUsages::RENDER_WORLD` on our Meshes.

We want to detect mouse/touch clicks on Nodes to "select" them (show an outline and perform other operations on them later on).

Today, we can't use `MeshPickingPlugin` because it relies on the CPU-side raycast which doesn't work because we don't store the Meshes in the `MAIN_WORLD`.

We tried making a custom backend, but it's supposed to be called every frame, even when there's no mouse click, which we can't do because it would lead to hundreds of API calls.

## What solution would you like?

Support asynchronous picking backends that can be executed only when needed like on mouse clicks, not on every frame.

## What alternative(s) have you considered?

We were considering the following:
- Make our custom backend trigger a raycast only when there's a mouse click, then repeat the `PointerHits` events a few frames so [`pointer_events`](https://github.com/bevyengine/bevy/blob/73970d0c1215ee2f290513da7aed2e619a77bd7b/crates/bevy_picking/src/events.rs#L418) thinks there has been a click.
- This doesn't work because the mouse click has already happened by the time [`pointer_events`](https://github.com/bevyengine/bevy/blob/73970d0c1215ee2f290513da7aed2e619a77bd7b/crates/bevy_picking/src/events.rs#L418) receives the `PointerHits`.
- Not using the picking of Bevy at all.
- Switch to a "color picking shader" method instead, as described in https://github.com/aevyrie/bevy_mod_picking/issues/3.

## Additional context

We tried doing something like this to do the raycast only after a mouse click:
```rs
#[allow(clippy::too_many_arguments)]
pub fn update_hits(
ray_map: Res,
picking_cameras: Query<&Camera>,
mut ray_cast: MeshRayCast,
mut output: EventWriter,
mouse_button_input: Res>,
) {
if !mouse_button_input.just_released(MouseButton::Left) {
return;
}

// Do the async raycast here.

// Send the event later.
commands.send_event(PointerHits::new(...));
}

// Repeat the PointerHits event to simulate the event being triggered for 2 frames.
fn repeat_pointer_hits(
mut pointer_hits: EventReader,
mut commands: Commands,
mut event_sent: Local,
) {
if *event_sent {
*event_sent = false;
return;
}

for ev in pointer_hits.read() {
println!("repeat_pointer_hits: Pointer hits: {:?}", ev.clone());
commands.send_event(ev.clone());
*event_sent = true;
}
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reading the MeshPickingPlugin flow and the pointer_events logic in crates/bevy_picking/src/events.rs, then trace how picking backends produce PointerHits. Done should allow a backend to perform work only when requested, including after a delayed asynchronous result, while still producing usable pointer events for selection.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
game-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.