google-deepmind / google-deepmind/lab
File resources leak: too many open files
- Dominant language
- C
- Stars
- 7.4k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
Consider the following code --- We generate 1000 episodes, each of which is made of 100 steps. For each episode a `deempind_lab.Lab` instance is created and closed.
It should run without an error, but the number of file descriptor keeps increasing until the process dies with an error: **too many open files** (or Segmentation Fault if the native code fails to open a file).
```python
import numpy as np
import deepmind_lab
from tqdm import tqdm
import tempfile
for i in tqdm(range(1000)):
env = deepmind_lab.Lab(
'nav_maze_static_01',
observations=[
'RGB_INTERLEAVED', 'DEBUG.CAMERA_INTERLEAVED.TOP_DOWN',
'DEBUG.POS.TRANS', 'DEBUG.POS.ROT', 'VEL.TRANS', 'VEL.ROT',
],
renderer='hardware',
)
env.reset()
action = np.zeros([7], dtype=np.intc)
for t in range(100):
env.step(action)
env.close()
with tempfile.TemporaryFile() as fp:
pass # too many open files?
```
In my linux system, it was killed at around 250 episodes; at the moment the number of open files (`lsof | grep 8922 | wc -l`) was approximately 4000. FYI, `ulimit -n` gives 1024.
```
$ python m.py
25%|████████ | 251/1000 [05:36<16:33, 1.33s/it]
[1] 8922 segmentation fault (core dumped) python m.py
```
### What happens?
The content of `lsof` is filled with:
```
...
python 8922 8929 (ID) DEL REG 0,5 384322126 /dev/zero
python 8922 8929 (ID) DEL REG 0,5 384322129 /dev/zero
python 8922 8929 (ID) DEL REG 0,5 384322148 /dev/zero
python 8922 8929 (ID) DEL REG 0,5 384322149 /dev/zero
...
```
as well as a number of
```
...
python 8922 8929 (ID) 811r REG 8,3 55390150 6423400 /home/(ID)/.cache/bazel/_bazel_(ID)/91edf13a2436a0b47d7f3009b4283215/execroot/org_deepmind_lab/bazel-out/k8-opt/genfiles/baselab/assets.pk3
python 8922 8929 (ID) 812r REG 8,3 195246 60031226 /home/(ID)/.cache/bazel/_bazel_(ID)/91edf13a2436a0b47d7f3009b4283215/execroot/org_deepmind_lab/bazel-out/k8-opt/genfiles/baselab/assets_bots.pk3
python 8922 8929 (ID) 813r REG 8,3 9662 60031225 /home/(ID)/.cache/bazel/_bazel_(ID)/91edf13a2436a0b47d7f3009b4283215/execroot/org_deepmind_lab/bazel-out/k8-opt/genfiles/baselab/assets_oa.pk3
python 8922 8929 (ID) 814r REG 8,3 371473 6815822 /home/(ID)/.cache/bazel/_bazel_(ID)/91edf13a2436a0b47d7f3009b4283215/execroot/org_deepmind_lab/bazel-out/k8-opt/genfiles/baselab/vm.pk3
...
```
One can easily see these lines are proliferative when comparing two snapshots of `lsof` outputs.
To sum, file descriptor of `/dev/zero` and resource files `(*.pk3)` are created and not **properly freed**. This would be an obvious resource leak. I haven't looked into where they are opened from.
P.S. For your information, the reason why I was trying to create new `Lab` instances every time instead of reusing one instance throughout the process includes #133 and #134.
Contributor guide
Assessment
This issue has not been assessed yet.