AusClimateService / AusClimateService/hazard_fire
Best-practice for zarr format data collections on linux filesystems
- Dominant language
- Jupyter Notebook
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
@matear & @e-peacey
RE: the conversation about how to best use `zarr` on `Gadi` I have my own issue here that I've not yet had the chance to make progress on. [write code to convert zarr collections to zarr-zipstore](https://github.com/Thomas-Moore-Creative/Climatology-generator-demo/issues/12 )
While I don't yet have any direct experience with using `zarr-ZipStore` my understanding is it doesn't appear to effect performance much ( this needs to be tested ) and it solves the `inode` problem that cloud optimised formats like `zarr` have on linux filesystems. Normally zarr looks like "one file per chunk" to a linux FS which is typically a much larger `inode` footprint than `netcdf`.
See: [https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore](https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore)
Contributor guide
No contributing guide indexed for this repository
Research direction
Read the linked issue about converting zarr collections to zarr-zipstore and the referenced zarr ZipStore documentation first. Establish whether ZipStore changes performance and resolves the inode footprint on Gadi's Linux filesystem; done means a tested recommendation or an agreed conversion implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100