HDFGroup / HDFGroup/hdf5

Chunk preallocation outside of dataset resize

Open
#3,229 3 comments 0 reactions 1 assignee Claimed by @mattjala View on GitHub
Component - C Library
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

I use a chunked dataset for incremental I/O.
When I need to append data, I enlarge the dataset. Chunks are automatically allocated as needed.

For performance, since I approximately know how much data will be appended, I would like to pre-allocate chunks so that enlarging the dataset will be (hopefully) cheaper.

But :

- Would `H5Dset_extent(did, newExtent);` be really cheaper ? I assume so, but I can't be sure.
- Is it even possible to preallocate chunks without a hack involving "extend then shrink the dataset" ?
- Is "extend then shrink the dataset" a solution to force the dataset to allocate chunks ? I can't find documentation about sparse allocation (that would make it useless) and auto-vacuum (that would ruin the benefit of enlarge+shrink) that would tell how "extra" chunks are automatically handled.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.