HDFGroup / HDFGroup/HDF.PInvoke

[HELP] Wiki and demo code about read ultra large dataset

Open
#186 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
87
Forks
31
PR merge metrics
No merged PRs in 30d

Description

Hi,

Many thanks for this great data format library.

I'm working on read a ultra large hdf5 dataset via HDF.PInvoke recently. Due to the reason of .NET array can not be allocated in size greater than 2GB, so if the data inside the hdf5 file its size is greater than 2GB, then the program crashed.

The dataset which its size less than 2GB could be read via the demo code in wiki [Cookbook : Datasets](https://github.com/HDFGroup/HDF.PInvoke/wiki/Cookbook-:-Datasets), and I guess i could read the dataset which its size greater than 2GB chunk by chunk via HDF.PInvoke, but after google search around it, i was not able to figure out how to implements such code correctly.

Here is the DEMO data in ultra large size that i currently working on: download the [P_1900 | Human Prostate](https://www.metabolomicsworkbench.org/data/DRCCMetadata.php?Mode=Study&StudyID=ST001857&StudyType=MS&ResultType=5) dataset from this repository.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.