google / google/grain

Grain workers allocates GPU memory

Open
#820 1 comment 2 reactions 1 assignee Claimed by @iindyk View on GitHub
type:performance
Dominant language
Python
Stars
779
Forks
86
Avg merge
2d 6h
Merged PRs (30d)
6

Description

When setting the GrainDataLoader worker_count > 1, each worker will use 500MB of GPU memory. When having many worker, this quickly depletes the amount of available memory, even if all processing is done on the CPU.

For example:

Image

I have tried to avoid this by preventing CUDA to be used for sub processes like this:

```
import os
import multiprocessing
if multiprocessing.parent_process() is not None:
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
os.environ["JAX_PLATFORMS"] = "cpu"
```

The root cause for this seems to be any reference to jax. When jax is imported, it seems to want to allocate some GPU memory. If we stick to numpy, this does not happen. We do need to use jax for resizing image/video tensors in a Transform operation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.