AusClimateService / AusClimateService/hazard_fire

Best-practice for zarr format data collections on linux filesystems

Open
#2 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

@matear & @e-peacey

RE: the conversation about how to best use `zarr` on `Gadi` I have my own issue here that I've not yet had the chance to make progress on. [write code to convert zarr collections to zarr-zipstore](https://github.com/Thomas-Moore-Creative/Climatology-generator-demo/issues/12 )

While I don't yet have any direct experience with using `zarr-ZipStore` my understanding is it doesn't appear to effect performance much ( this needs to be tested ) and it solves the `inode` problem that cloud optimised formats like `zarr` have on linux filesystems. Normally zarr looks like "one file per chunk" to a linux FS which is typically a much larger `inode` footprint than `netcdf`.

See: [https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore](https://zarr.readthedocs.io/en/stable/api/storage.html#zarr.storage.ZipStore)

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the linked issue about converting zarr collections to zarr-zipstore and the referenced zarr ZipStore documentation first. Establish whether ZipStore changes performance and resolves the inode footprint on Gadi's Linux filesystem; done means a tested recommendation or an agreed conversion implementation scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.