google-deepmind / google-deepmind/torch-hdf5

Unable to read the "master" hdf5 file after linking smaller hdf5 files to it

Open
#111 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
241
Forks
125
PR merge metrics
No merged PRs in 30d

Description

I have a couple of small (~20-200GBs) hdf5 files that I link together to get a single hdf5 file (as described [here](https://stackoverflow.com/a/18527687/2838606)). However, I cannot read this larger, master, hdf5 file anymore using torch-hdf5 package when using multiple threads but I can read the smaller files. I wonder if the hdf5 package has issues reading "linked" hdf5 files?

Contributor guide

Open the contributing guide

Research direction

The issue names no repository files or tests. Start by reproducing the linked HDF5 setup with torch-hdf5 under multiple threads, then compare reads from the linked master file with the smaller files; done means identifying whether the failure is specific to linked files and documenting a reproducible result.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.