google-deepmind / google-deepmind/torch-hdf5

Iterating through data

Open
#31 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Lua
Stars
241
Forks
125
PR merge metrics
No merged PRs in 30d

Description

I am using torch-hdf5 to exchange big chunks of data between Python and Lua. I want to iterate through the data without loading everything first. I am able to do it with the code below but I am accessing directly (undocumented) member variables. Is there a better way?

``` lua
h5 = hdf5.open('file.h5', 'r')
for k,c in pairs(h5._rootGroup._children) do
local d = h5:read(k):all()
end
```

Contributor guide

Open the contributing guide

Research direction

Start with the public hdf5.open and h5:read APIs shown in the issue, then inspect how datasets are exposed without relying on _rootGroup._children. Determine the supported iteration behavior and document or add an API that avoids loading the full dataset; done means the Lua example can iterate through the data using public interfaces.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.