google-deepmind / google-deepmind/mujoco
Add a maximum ray-casting distance for `rangefinder` sensors
- Dominant language
- C++
- Stars
- 15.2k
- Forks
- 1.8k
- Avg merge
- 10d 16h
- Merged PRs (30d)
- 25
Description
# The feature, motivation and pitch
Request to add a way for a `rangefinder` sensor to limit ray casting to a maximum detection distance.
## Current Implementation
Currently, the `rangefinder`'s `cutoff` parameter only performs output post-processing. MuJoCo first casts the ray with no maximum distance, finds the nearest intersection, then clips the result.
The clipping occurs as follows, where $d$ is intersection distance and $C$ is the cutoff distance.
| `cutoff` | Condition | `rangefinder` Sensor Output |
| --- | --- | --- |
| `C == 0` (default) | Intersection found | d |
| `C == 0` (default) | No intersection found | -1 |
| `C > 0` | Intersection found at `d <= C` | d |
| `C > 0` | Intersection found at `d > C` | C |
| `C > 0` | No intersection found | -1 |
I.e., for a positive cutoff,
$$
f \left( d, C \right) =
\begin{cases}
\min\left( d, C \right), & \text{intersection found} \\
-1, & \text{no intersection found}
\end{cases}.
$$
## Desired Behavior
I am requesting that the behavior of the sensor when a _positive `cutoff`_ is provided be modified as follows.
$$
f \left( d, C \right) =
\begin{cases}
d, & 0 \leq d \leq C \\
-1, & \text{otherwise}
\end{cases}
$$
| Condition | `rangefinder` Sensor Output |
| --- | --- |
| Intersection found at `d <= C` | d |
| Intersection found at `d > C` | -1 |
| No intersection found | -1 |
The behavior should not change for the default `cutoff == 0`.
Site, perspective-camera, and orthographic-camera `rangefinder`s should all use their existing miss representations for their respective output fields for out-of-range intersections. The cutoff should apply to ray distance rather than camera-plane depth for perspective cameras.
## Rationale
### Real-wold relevance
The current behavior does not accurately represent many real-world distance sensors.
Time-of-flight, LIDAR, and ultrasonic distance sensors all have a maximum effective range, and generally will not report a value beyond that cutoff distance. A sensor with a maximum detection range of 2-meters generally cannot detect an object at 5-meters. Reporting a clipped measurement of 2-meters incorrectly implies a detection when there was not one. Reporting `-1`` represents the missing measurement more clearly.
Stereo depth cameras are a little bit more complicated. The current approach better models a stereo camera that clamps far-away estimates to a maximum distance. However, the requested behavior is more apt for a stereo algorithm that marks far-away pixels as invalid.
The requested behavior is therefore a better model for sensors with a hard detection limit, but not for every distance sensor.
### Simulation performance
Without a maximum distance, MuJoCo still checks geometry beyond the sensor's useful range. These checks can be expensive, especially for complex shapes.
A maximum distance would let MuJoCo skip geometry beyond the cutoff. After finding a closer hit, `geom`s/intersections beyond that distance could also be skipped.
`mj_multiRay()` already uses its cutoff to skip obviously distant `geom`s, but a final check is still needed to reject hits beyond the exact limit. Site and orthographic-camera `rangefinder`s use `mj_ray()`, which does not currently support a maximum distance.
### Arguments against
There are valid reasons to preserve the current behavior:
- Some physical sensors or software pipelines saturate their output while retaining a valid detection.
- Simulation-only consumers may need to distinguish a distant object from no object.
- Some consumers may prefer the current finite, capped semantics.
- The current output is continuous at the cutoff, while the requested output changes abruptly from `C` to `-1`.
Users could reproduce the current distance behavior with an unlimited rangefinder and clipping in their application.
For these reasons, it may be worthwhile to support maximum detection distance and output clipping as separate behaviors.
# Alternatives
## Add a separate `maxdist` attribute
Add a `rangefinder`-specific `maxdist` attribute for maximum detection distance and retain `cutoff` as output clipping.
This would preserve existing behavior and distinguish the two different physical concepts:
- `maxdist` - maximum detection distance, beyond which there is no valid return
- `cutoff` - output saturation, which clips a successfully measured value.
## Create a user or plugin sensor with the `mj_multiRay()` `cutoff` parameter
This would address individual models/applications, but is duplicating the functionality of the built-in `rangefinder`. Also, the existing `mj_multiRay()` function does not implement all possible optimizations and the `mj_ray()` function has no equivalent `cutoff` parameter.
# Additional context
- MuJoCo already gives the `cutoff` attribute [sensor-specific semantics for collision sensors](https://mujoco.readthedocs.io/en/stable/XMLreference.html#collision-sensors), so interpreting it differently for a particular sensor is not outlandish.
Contributor guide
Research direction
Start with the mj_multiRay() and mj_ray() entry points mentioned in the issue, then trace how rangefinder sensors handle cutoff values for site, perspective-camera, and orthographic-camera sensors. Done means positive cutoffs reject intersections beyond the limit while cutoff == 0 keeps existing behavior, with each sensor using its existing miss representation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- computer-graphics, robotics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100