google-deepmind / google-deepmind/torch-hdf5

Out of IDs after reading many files

Open
#63 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Lua
Stars
241
Forks
125
PR merge metrics
No merged PRs in 30d

Description

I'm using many data files in hdf5 format to train a neural network. After running for many epochs over a few hours, it crashes with an error

```
HDF5-DIAG: Error detected in HDF5 (1.8.16) thread 140335388788608:
#000: H5F.c line 608 in H5Fopen(): unable to atomize file handle
major: Object atom
minor: Unable to register new atom
#001: H5I.c line 921 in H5I_register(): no IDs available in type
major: Object atom
minor: Out of IDs for group
```

It seems to be a known(?) bug, and exists in both 1.8.14 and 1.8.16
https://stackoverflow.com/questions/35522633/hdf5-consumes-all-resource-ids-for-dataspaces-and-exits-c-api

I can reproduce it with this if I just let it run for a while (to be precise, around 2^24 = 16777216 iterations)

```
require 'hdf5'
require 'xlua'
local N = 20000000
local n = '/tmp/test.h5'
local f = hdf5.open(n, 'w')
f:write('/data', torch.rand(1))
f:close()

for i=1,N do
local f = hdf5.open(n, 'r')
f:read('/data'):all()
f:close()
xlua.progress(i, N)
end
```

Any ideas? Should I just not use hdf5?

Contributor guide

Open the contributing guide

Research direction

Start by running the provided Lua reproduction with repeated hdf5.open, read, and close calls, then inspect how those operations release HDF5 identifiers. The issue is resolved when the loop can complete without the HDF5 "Out of IDs for group" error; no project file or test is named, so locating the relevant implementation and adding coverage will require investigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.