bevyengine / bevyengine/bevy

Tracking issue: Parallax mapping improvements

Open
#8,389 1 comment 0 reactions 0 assignees View on GitHub
A-Rendering C-Feature C-Tracking-Issue
Dominant language
Rust
Stars
48.2k
Forks
4.8k
Avg merge
3d 16h
Merged PRs (30d)
171

Description

This issue tracks the progress on the "Future works" section of the parallax mapping PR #5928.

Except distance fading and depth buffer update, bevy's parallax mapping implementation is on par with similar features from other general game engines. But it is far from being state of the art.

### Contributing

Want to check off one of those checkboxes? Nothing easier!

1. [Open a feature request][open-feature-request] with, as title, "Add \ for parallax mapping", you can copy/paste the description if you feel lazy!
2. Implement the feature and open a PR!

All related issues will be listed in this tracking issue.

### Low hanging fruits

Some features weren't added in the original parallax mapping PR in order to avoid scope creep, not because they are particularly hard to implement. Those are low hanging fruits that could be implemented.

- [ ] **Distance fading**: user-configurable limit on the view range in which parallax mapping is applied. Parallaxing doesn't make sense on distant objects as the difference of view position is small enough to be insignificant, yet it still incurs a significant cost. [This video][fading-video] motivates distance fading.
- [ ] **Improved mipmap sampling**: Mipmapping is another way to reduce perf cost of parallaxing on distant materials. The current implementation chooses to sample at LOD level 0 for performance and practicality. It also works around the naga default Dx12 compiler. However, there is nothing preventing us from pre-computing a gradient and using this set gradient when sampling the depth map. [This paper discusses the perf characteristics of various sampling methods in POM][optimization-tips].
- [ ] **Offset limiting method**: This is the ancestor of steep parallax mapping. It is not as accurate, but has the advantage of *requiring no iteration*, only a simple sampling of the depth map. Therefore, it is a great method to use for weak devices. It is described in [the original article used as reference for the implementation][sunandblackcat] and [an original paper][offset-limiting-paper].

### Important but requires moderate efforts

Those are features that *would be really nice to have* but are not implemented because it would require a somewhat consequent time investment.

- [ ] **Update the depth buffer**: Updating the depth buffer according to the depth map would allow accurate silhouetting in intersecting geometry. it would also allow shadows cast into and from parallaxed surfaces to follow the depth-mapped surface, rather than just the mesh surface. However, it's both costly (need to run parallaxing for all source of light!) and error prone (need to compute the additional depth relative to viewport). #8486
- [ ] **Depth maps glTF extension**: We currently do not load depth maps from glTF because glTF doesn't have such a feature! [A proposal is open but seems heavily stalled][gltf-paramap]. However, glTF is extensible and you can add a depth map through the `extras` field on materials. This would require:
1. Changing the glTF loading code from `bevy_gltf` to support that extension.
2. Defining the extension formally so that it's possible to modify authoring software to include the depth information in the glTFs it exports (say for example through the [blender gltf-io plugin][blender-gltf]).
3. Modify at least one authoring tool to support such an export option.
- [X] **Generate the depth maps from normal maps**: A work around the lack of depth map is to simply deduce it from the normal map. Implemented in .

### More complex but _fun_ improvements

Finally, we are left with the "state of the art" features. The last improvements in parallax mapping. Although, since parallax mapping is an old and tried algorithm, "state of the art" is already quite old. Regardless, implementing those would elevate bevy's implementation of parallax mapping as one of the very best.

- [ ] **Cone mapping / relaxed cone stepping method**: This [GPU gems article][gpugems-conemap] explains a variant of parallax mapping using additional information stored in the depth map texture to greatly reduce the cost of the initial ray-marching step, and improved accuracy. It is known to be used in [SimCity 2013][simcity-cone]. It seems not that hard to implement, however would require pre-processing the depth map asset.
- [ ] **Ray-quadratic distance**: This barbaric name hides an extremely wishable feature: extending parallax mapping to curved surfaces. The current implementation doesn't play well with curvature. By adding quadratic surface parameter attributes to mesh vertices, it is possible to compute a more accurate ray that account for mesh surfaces and allow "missing" the material when the parallaxed surface is not hit, for true silhouetting. [This paper describes in detail the method][rayqd].
- [ ] **Quadtree displacement mapping method**: is another parallax mapping variant that aims to reduce the ray-marching steps, based on carefully built mipmaps. However, the performance gains [are contested][optimization-tips]. It is discussed in [this series of slides][qdm].

[rayqd]: https://www.inf.ufrgs.br/~oliveira/pubs_files/Oliveira_Policarpo_RP-351_Jan_2005.pdf
[qdm]: https://www.gamedevs.org/uploads/quadtree-displacement-mapping-with-height-blending.pdf
[simcity-cone]: https://community.simtropolis.com/omnibus/other-games/building-and-rendering-simcity-2013-r247/
[gpugems-conemap]: https://developer.nvidia.com/gpugems/gpugems3/part-iii-rendering/chapter-18-relaxed-cone-stepping-relief-mapping
[rust-fourier]: https://github.com/mpizenberg/fft2d/blob/main/examples/normal_integration.rs
[stannum-fourier]: https://stannum.io/blog/0IwyJ-
[houdini-poisson]: https://houdinigubbins.wordpress.com/2019/08/09/from-normal-to-height/
[rust-screen-poisson]: https://lib.rs/crates/poisson_reconstruction
[reddit-n2d]: https://old.reddit.com/r/gamedev/comments/fffskm/convert_normal_map_to_displacement_map/
[nvidia-slides-n2d]: https://developer.download.nvidia.com/assets/gamedev/docs/nmap2displacement.pdf
[blender-gltf]: https://projects.blender.org/blender/blender-addons/src/branch/main/io_scene_gltf2
[gltf-paramap]: https://github.com/KhronosGroup/glTF/pull/2196
[sunandblackcat]: https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28
[offset-limiting-paper]: https://raw.githubusercontent.com/marcusstenbeck/tncg14-parallax-mapping/master/documents/Parallax%20Mapping%20with%20Offset%20Limiting%20-%20A%20Per-Pixel%20Approximation%20of%20Uneven%20Surfaces.pdf
[open-feature-request]: https://github.com/bevyengine/bevy/issues/new?assignees=&labels=C-Enhancement%2C+S-Needs-Triage&template=feature_request.md&title=
[fading-video]: https://www.youtube.com/watch?v=8hThP-Yni_o
[optimization-tips]: https://www.diva-portal.org/smash/get/diva2:831762/FULLTEXT01.pdf

Contributor guide

Open the contributing guide

Research direction

Start by reading parallax mapping PR #5928 and choose one unchecked improvement from this tracking issue. Open a feature request using the specified title, then implement that feature and verify it before opening a pull request; completion depends on the selected improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
computer-graphics, game-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.