[Bug] Rounding issues in TerrainLayer

Description

I am attempting to visualize high resolution terrain data using the TerrainLayer, but it seems that at higher zoom levels the layer is unable to correctly render the tiles. The cause of this seems to be a rounding error due to use of float32.

The problem starts at zoom level 17 in my case, but I guess this will depend on the cartesian offset.

Wireframe showing the rounding error.

While debugging this issue, the first problem was actually found in the @loaders.gl / parse_terrain.ts /getMeshAttributes where the 32 bit resolution forces multiple x and y coordinates into the same values. I overcame this issue by modifying the code to use zero based bounds and a scaling factor, then communicating these parameters with the generated mesh so that the creation of the SimpleMeshLayers in TerrainLayer.renderSubLayers can compensate through getPosition and getScale.

Now I can verify that the mesh generated by the TerrainLoader is ok, but the rendering is still messy like in the picture above, and i assume this is caused by later mesh processing. I have spent a few days digging, but I'm still unable to pinpoint exactly where the error lies. Suspecting it could be mesh_layers/utils/matrix.ts ...

Flavors

• React
• Python/Jupyter notebook
• MapboxLayer
• GoogleMapsOverlay
• CartoLayer
• DeckLayer/DeckRenderer for ArcGIS

Expected Behavior

At a minimum, the limitations in the TerrainLayer and TerrainLoader should be documented, e.g. max supported zoom level..

I would love to get some pointers on how to solve this issue in my current project short term, and of course get a long term solution in place in an upcoming release.

Steps to Reproduce

• Using a elevation tile source with tiles for zoom level > 17
• meshMaxError = 0
• wireframe = true

Problem should be visually apparent

Environment

• Framework version: deck.gl@8.8.13
• Browser: Chrome 108.0.5359.125
• OS: Windows 11

Logs

No response

Start by reproducing the issue with an elevation tile source above zoom 17, meshMaxError=0, and wireframe=true. Inspect @loaders.gl/parse_terrain.ts, especially getMeshAttributes, and then mesh_layers/utils/matrix.ts, since the report suspects later mesh processing. Done means TerrainLayer renders high-zoom terrain correctly and its supported zoom limitations are documented.