dfm / dfm/corner.py

Memory leak with corner.corner()

Open
#171 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
576
Forks
234
PR merge metrics
No merged PRs in 30d

Description

I'm using the corner package, version 2.0.1 to produce some corner plots after importing matrices stored in text files.
My laptop has 16 GB RAM + 16 GB SWAP. The cluster at my institute has ~128 GB of memory.
Matrices to be loaded may vary from 1.2e6 x 14 elements up to 13e6 x 26 elements and they are loaded and stored through np.loadtxt() without any problem.
I load matrices and produce the corner plots within a for cycle so that at the following cycle step the previous matrix is overwritten by the new one. To reduce memory consumption I also delete both the stored matrix and the figure variable produced by corner.corner() with the following commands

  • del
  • gc.collect()
    The number of panels in the corner plots may vary from 105 to 351.

Monitoring the memory status through htop the memory consumption starts increasing when corner.corner(matrix) is running and for the biggest matrices even at the cluster shell I got 'Killed' when memory is exhausted.

The matrices I'm dealing with are large, but not insanely large. Could you have a check to this memory issue?

Best regards,
Andrea

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing corner.corner(matrix) with the large matrix sizes and panel counts described in the issue, using np.loadtxt() and monitoring memory with htop. Compare memory before and after deleting the matrix and figure and running gc.collect(); done means the source of the retained memory is identified and the issue is resolved without exhaustion.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
data-visualization, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.