hdf5 chunk defaults and handling
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 150
- Forks
- 23
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 2
Description
The current behaviour when reading a chunked file is somewhat surprising (to me). If one reads this variable:
float UM_m01s02i205_vn1106(time, latitude, longitude) ;
# skip uninteresting attributes for this issue
UM_m01s02i205_vn1106:_Storage = "chunked" ;
UM_m01s02i205_vn1106:_ChunkSizes = 1, 1920, 2560 ;
I see the following unexpected result:
In [30]: g = cf.read('double-chunking-testc.nc')[0]
In [31]: g.data.nc_hdf5_chunksizes()
Out[31]: ()
This is not a bug, insofar as it is the expected behaviour of the code - by construction cf-python currently doesn't remember HDF chunksizes from the read.
Should it? If so, it could be done, possibly with certain caveats on when that's a sensible thing to do, and it may well forget them when certain operations are applied (e.g. when aggregating files with different HDF chunks, when subspacing, when adding/removing/transposing dimensions, etc.).
Another V4.0 issue!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the example with cf.read('double-chunking-testc.nc')[0] and inspect the behavior of nc_hdf5_chunksizes(), which currently returns an empty tuple. The issue needs a decision on whether chunk sizes should be retained and which operations should discard them; completion is not defined until those expectations are specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100