google / google/grain

AttributeError: 'NoneType' object has no attribute 'mmap'

Open
#398 4 comments 6 reactions 0 assignees View on GitHub
type:bug
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

This seems to happen at shutdown in any data pipeline that has NumPy arrays. Here is the full stacktrace:

```
INFO:absl:Process 0 exiting.
INFO:absl:Processing complete for process with worker_index 0
INFO:absl:Grain pool is exiting.
INFO:absl:Shutting down multiprocessing system.
INFO:absl:Shutting down multiprocessing system.
Exception ignored in:
Traceback (most recent call last):
File "/home/black/micromamba/envs/trainpi/lib/python3.10/site-packages/grain/_src/python/shared_memory_array.py", line 139, in __del__
AttributeError: 'NoneType' object has no attribute 'mmap'
/home/black/micromamba/envs/trainpi/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
```

Even if it's not an actual problem, it's a bit annoying because it overwhelms the logging output when you have many workers.

Here's the simplest possible repro:
```python
import grain.python as grain
import numpy as np
import logging
logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
class DataSource:
def __len__(self):
return 10

def __getitem__(self, idx):
return np.zeros(1)

source = DataSource()
sampler = grain.IndexSampler(
num_records=len(source),
num_epochs=1,
shard_options=grain.NoSharding(),
shuffle=False
)
loader = grain.DataLoader(
data_source=source,
sampler=sampler,
worker_count=1,
)

for batch in loader:
pass
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.