Roadmap for WebGL visualization of Icechunk stores.

After meeting with @dzole0311 the previous week regarding his requirements for accessing HRRR data fire visualization and @hanbyul-here 's recent work on https://github.com/developmentseed/numeric-data-layer we have some momentum and converging efforts on loading and visualizing Icechunk data in a WebGL context.

We'd like to reach an eventual state where we have WASM Icechunk API that WebGL can use via shared zero-copy memory similar to the vector equivalent of employing https://github.com/kylebarron/parquet-wasm/ with DeckGL. But reaching this goal will require a significant investment in time and building some foundational technology so I propose we take a staged approach so that we can build functioning applications quickly and assess the performance ROI for each stage of development. Specifically for @dzole0311 use case of rendering HRRR wind speed and direction information I think we may want to consider this long term plan

Traditional tile based approach

I'll make a separate ticket that outlines the steps we'll need to execute this specifically for the HRRR data, but most of the tech is ready for this today. We can build a simple tiling proxy for an Icechunk store (ideally one that follows WOZ guidelines) using titiler-multidim or xpublish and visualize it using a Deck.gl TileLayer or via MapboxGL depending upon the context.

Intermediate JS memory representation

There are excellent existing libraries for reading Zarr data directly in JS zarrita.js and zarr.js. These libraries only support native Zarr stores. To support virtualized data we need to use Icechunk rather than native Zarr. Because the core Icechunk library is written is Rust we can leverage wasm-bindgen to build WASM bindings for Icehunk and create a key value Zarr retrieval JS wrapper on top of these bindings.

This allows us to retrieve bytes from virtualized datasets stored in Icechunk through WebGL shaders with libraries like https://github.com/carbonplan/maps or https://github.com/carderne/zarr-gl.

There has been some investigation around this at https://github.com/earth-mover/icechunk/issues/356 but I don't believe this a current priority for Earthmover so we'd likely need to tackle this ourselves and request updates to the Icechunk library to make WASM compilation possible.

This is a big step forward but IIUC we are still using the Icechunk bytes->WASM->JS->WebGL path that requires a the potentially inefficient step of moving raw data out of WASM memory into a JS memory representation before passing it into WebGL memory for rendering. To reach our ultimate zero copy goal we want a ...

Native Icechunk WebGL library

This ultimate goal requires several moving pieces to be developed and integrated

1. Leverage the Icechunk WASM binding from above and generate WASM bindings for https://github.com/LDeakin/zarrs_icechunk so that we can perform Zarr API key value retrieval in WASM memory space.
2. Build our own WebGL shader/rendering library (with interoperability for MapboxGL and Deck.gl) that uses zero copy WASM memory buffers through WebGL buffer data without an intermediate JS array representation.

While we have the technical capabilities to tackle #1 I lack even a basic enough knowledge on the WebGL side to understand what skills we'd need to build out this library or who would be best to collaborate with in order approach this.

Again, this is just a high level roadmap ticket to describe how we might progress to an eventual zero copy Icechunk in browser rendering experience. I'll create an additional ticket with a concrete objective doing the "Traditional tile based approach" for the AWS HRRR data to support fire visualization work for VEDA in the short term.

No repository files or tests are identified. Start by reviewing Icechunk issue #356 and the linked numeric-data-layer, wasm-bindgen, zarrs_icechunk, Deck.gl, and WebGL references; done would be a concrete, scoped follow-up for one staged approach rather than this high-level roadmap.