Add Ambient Occlusion layer to enhance 3d scene depth perception

Prototype

To enhance 3d scene perception (similarly to fog proposed in https://github.com/mapbox/mapbox-gl-js/issues/10205) we could introduce the use of an Ambient Occlusion (AO) layer, also called Sky View Factor. It's an indirect lighting term representing how much a point in space sees the sky for a surface on the ground and can be complementary to direct lighting (hillshading). It is called indirect lighting as it represents how much light bounces from other locations and increases the perceived brightness of a surface, compared to more direct or local lighting only shading with information gathered from the currently shaded point. Since ambient occlusion has a more global effect, areas that are harder to reach from the skylight will become darker compared to areas that are very exposed to it.

AO better retains depth cues and increases overall depth perception of a scene. Since this term varies based on how occluded a shaded point is, it allows for an understanding of how steep, narrow, and open terrain areas are and better layers terrain features between each other. The main effect is pushing some areas in the background while retaining others in the foreground. This improves landscape understanding which isn't easily representable with typical Hillshading, as it suffers from flattening everything at the same level of depth.

We can use a line-sweep method from this paper to sweep the DEM in a few directions, and for each of the sweeps build a list of horizon points and applies an occlusion factor which is then averaged for all sweep directions. Here are some results from prototyping:

To create an AO term we've explored two different techniques: standard deviation with local hypsometric tints and the line sweeps which gave the best results in the prototype. I also have the intuition that the line sweeps could be later used to add shadows, as a single sweep with a reduced falloff distance could approximate shadows, as seen here:

Design considerations

• How to control AO intensity and color
• How does AO blend with hillshade
• What blending modes are necessary for the introduction of such a layer

Risk

Since this lighting term is global it requires tile neighboring information. We can have large backfill lookup on neighboring tiles (100px gave good results as proposed by @arindam1993 ). Another unexplored option may be to build and store the 2d elevation convex hull result at tile boundaries. Though, the term is constant enough that it could justify server side implementation.