[Feature Request] Allocate chunks without writing via H5Dwrite_chunk or a new function
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
**Is your feature request related to a problem? Please describe.**
I would like to allocate space for chunks within a chunked dataset without actually writing the data. While `H5Pset_alloc_time` allows this to be done on a dataset scale, this does not allow the sizes or `filter_mask` of individual chunks to be specified.
The utility of this would be to separate the layout and data writing stages of HDF5 file creation. This would also allow chunks to be allocated within the file in an order distinct from the default order. For example, they could be in the order of Morton z-curve.
The ability to manipulate the chunk storage order or prescribe the chunk locations would increase compatibility with foreign I/O or APIs.
**Describe the solution you'd like**
One solution would be to modify `H5Dwrite_chunk` to accept a null pointer for the `buf` argument. A null pointer for `buf` indicates that no writing will be done. Perhaps the fill value or fill time properties could apply here although it is unclear how that might interact with the filters. My suggestion is that the initial implementation make no guarantees about the contents of the chunk other than its size.
**Describe alternatives you've considered**
1. Another solution would be to create a new API function called `H5Dalloc_chunk` that is similar to `H5Dwrite_chunk` but does not accept a `buf` argument.
2. For chunk ordering, a new property that influences chunk order could be created
**Additional context**
The proposed Zarr shard specification describes a method for storing chunks in a file. The chunk index is a simple sequence of file offsets and number of bytes. Some applications may prefer a certain order, such as Morton order.
https://zarr.dev/zeps/draft/ZEP0002.html
Contributor guide
Assessment
This issue has not been assessed yet.