google-deepmind / google-deepmind/torch-hdf5
Iterating through data
- Dominant language
- Lua
- Stars
- 241
- Forks
- 125
- PR merge metrics
- No merged PRs in 30d
Description
I am using torch-hdf5 to exchange big chunks of data between Python and Lua. I want to iterate through the data without loading everything first. I am able to do it with the code below but I am accessing directly (undocumented) member variables. Is there a better way?
``` lua
h5 = hdf5.open('file.h5', 'r')
for k,c in pairs(h5._rootGroup._children) do
local d = h5:read(k):all()
end
```
Contributor guide
Research direction
Start with the public hdf5.open and h5:read APIs shown in the issue, then inspect how datasets are exposed without relying on _rootGroup._children. Determine the supported iteration behavior and document or add an API that avoids loading the full dataset; done means the Lua example can iterate through the data using public interfaces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100