inducer / inducer/pyopencl

Possible memory leak when collecting events for profiling

Open
#625 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
1.2k
Forks
247
Avg merge
7h 2m
Merged PRs (30d)
6

Description

**Describe the bug**
Very large (host) memory consumption has been observed when running OpenCL application in profiling mode.
Example: Processing 10000 4Mpix images (int32) with ~6 kernels per image on a nvidia Tesla A40 gets (OOM-) killed on a computer with 200GB of memory. The computer could host all images, uncompressed, in memory.

I used the `tracemalloc` tool from Python on the application without noticeable leak (at the Python level) indicating that the leak was from malloc performed outside the scope of Python. I investigated a possible leak coming from `HDF5` via the `h5py` since all data were read and written in this format. but this was not the case.

When profiling is disabled, the memory consumption does not exceed few percent of the total memory.

**To Reproduce**
Investigated in:
https://github.com/silx-kit/pyFAI/pull/1744

**Expected behavior**
A memory leak is expected from keeping the list of all events, but should not exceed 3.4 MB for 60000 kernels (when stored as 2-namedtuple)

**Environment (please complete the following information):**
- OS: Linux Debian11 + Ubuntu 20.04
- ICD Loader and version: ocl-icd 2.2.14-2
- ICD and version: Nvidia 470.141.03-1~deb11u1
- CPU/GPU: Nvidia Titan V & Tesla A40
- Python version: 3.9 + 3.9
- PyOpenCL version: 2021, 1, 2 + 2021, 2, 13

**Additional context**
The list of event is handled at https://github.com/silx-kit/silx/blob/master/src/silx/opencl/processing.py#L288

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.