Write through cache for incremental I/O
- Dominant language
- C
- Stars
- 988
- Forks
- 355
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 12
Description
Let's assume an unlimited chunked dataset.
From time to time, I have to perform a bulk session of big writes of various sizes, (not necessarily equal to the chunk size), that do not need to be processed and could be written as raw data, without any filters. Bypassing the cache looks like a promising performance improvement but using [H5Dwrite_chunk](https://docs.hdfgroup.org/hdf5/develop/group___h5_d.html#title35) would be difficult for the following reasons :
1. depending on the chunk size, one write might not fill a chunk, so we have to read the destination chunk (H5Dread_chunk), before updating it and rewriting it as a single entity
2. when appending data, the destination chunk might not yet exist, so we would first check if a dummy non-raw write is required to force its allocation so that it can be addressed by `H5Dread_chunk`
3. depending on the chunk size, one write might span on several chunks, so we would have to manually loop over 1. and 2.
Ideally for that scenario (not considering the HDF5 design), there would be some H5DwriteRaw(), or a new data transfer property, that would automatically :
- allocate and raw-write whole chunks that fit in the submitted data
- for smaller writes or edge data (regarding chunk boundaries), keep a single chunk in cache to avoid raw-reading it at each update that would still not fill it completely
- invalidate regular cache as necessary
- support write offset not necessarily at the end of the file, nor on chunk boundaries
Contributor guide
Assessment
This issue has not been assessed yet.