HDFGroup / HDFGroup/hdf5

Write through cache for incremental I/O

Open
#3,230 1 comment 0 reactions 1 assignee Claimed by @fortnern View on GitHub
Component - C Library
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

Let's assume an unlimited chunked dataset.
From time to time, I have to perform a bulk session of big writes of various sizes, (not necessarily equal to the chunk size), that do not need to be processed and could be written as raw data, without any filters. Bypassing the cache looks like a promising performance improvement but using [H5Dwrite_chunk](https://docs.hdfgroup.org/hdf5/develop/group___h5_d.html#title35) would be difficult for the following reasons :

1. depending on the chunk size, one write might not fill a chunk, so we have to read the destination chunk (H5Dread_chunk), before updating it and rewriting it as a single entity
2. when appending data, the destination chunk might not yet exist, so we would first check if a dummy non-raw write is required to force its allocation so that it can be addressed by `H5Dread_chunk`
3. depending on the chunk size, one write might span on several chunks, so we would have to manually loop over 1. and 2.

Ideally for that scenario (not considering the HDF5 design), there would be some H5DwriteRaw(), or a new data transfer property, that would automatically :
- allocate and raw-write whole chunks that fit in the submitted data
- for smaller writes or edge data (regarding chunk boundaries), keep a single chunk in cache to avoid raw-reading it at each update that would still not fill it completely
- invalidate regular cache as necessary
- support write offset not necessarily at the end of the file, nor on chunk boundaries

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.