HDFGroup / HDFGroup/hdf5

H5Dread_chunk with partial reads

Open
#6,292 15 comments 0 reactions 2 assignees Claimed by @fortnern View on GitHub
Dominant language
C
Stars
988
Forks
355
Avg merge
4d 2h
Merged PRs (30d)
12

Description

I would love to be able to do a "partial read" of my raw chunk.
Some compression algorithm allow multi-level decompression, so doing a partial read, would give us information like:
1. what part of the chunk do we actually need to read to decode at a certain level.

I understand that this breaks some of the models in HDF5, but for certain very large dataset use cases, this kind of "multi-level reading" really does help.

# Describe the solution you'd like
Being able to provide a:
* offset within the chunk
* size of read

to read at the raw level.

# Describe alternatives you've considered
Saving the data in multiple resolutions.
That isn't fun now is it ;).

It just becomes hard to manage

# Additional context
I thought of doing this with the direct file pointer, but in certain cases, such as virtual dataset, it doesn't always exist.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.