Scoping Cloud Native Options for ATL11

User moving to MAAP are utilizing ATL11 data in a workflow. There are some potential opportunities to improve the data io for the existing algorithm through cloud optimization and rebuilding of spatial index methodologies.

Starting with a basic assessment of the existing product which can be potentially modified.
Following https://guide.cloudnativegeo.org/cloud-optimized-netcdf4-hdf5/#cloud-optimized-hdfnetcdf-checklist

A basic read of one granule ATL11_138712_0331_007_04.h5

h5stat -S

Filename: ATL11_138712_0331_007_04.h5
File space management strategy: H5F_FSPACE_STRATEGY_FSM_AGGR
File space page size: 4096 bytes
Summary of file space information:
  File metadata: 1036268 bytes
  Raw data: 62237486 bytes
  Amount/Percent of tracked free space: 0 bytes/0.0%
  Unaccounted space: 52367 bytes
Total space: 63326121 bytes

Page size might be ok, but file space strategy is not H5F_FSPACE_STRATEGY_PAGE

Checking the chunking of a randomish DATASET

DATASET "bsnow_conf" {
            DATATYPE  H5T_STD_I8LE
            DATASPACE  SIMPLE { ( 17902, 29 ) / ( 17902, 29 ) }
            STORAGE_LAYOUT {
               CHUNKED ( 2238, 8 )
               SIZE 41479 (12.516:1 COMPRESSION)
            }
            FILTERS {
               COMPRESSION DEFLATE { LEVEL 6 }
            }

2238 X 8 X 1 = 17904 bytes per chunk , or 17kb, which is below the min recommendation of 100kb, and not well aligned with Page Size.

Next recommended step is to explore all the variables in more depth using vzviz
A relevant example being ATL06 which is the precursor to ATL11 https://github.com/virtual-zarr/atl06-manifest-explorer

So far this analysis does not cover the spatial index used to identify which files/chunks to open.

Start by reviewing the cloud-native NetCDF4/HDF5 checklist and the existing ATL11 granule with h5stat. Use vzviz and the ATL06 manifest explorer as references to assess all variables, chunking, and the spatial index. Done means documenting the cloud-optimization opportunities and the remaining spatial-index analysis.