High memory usage when printing arrays with many dimensions
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 32.8k
- Forks
- 12.8k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 197
Description
NumPy has surprisingly high memory usage for printing arrays with many dimensions.
Reproducing code example:
import numpy as np
x = np.ones([2]*25)
str(x)
On my Mac, this script has a peak memory usage of 9.3GB and it takes 298s to run.
This array takes 268MB in memory, and its printed form as measured by len(str(x)) is 570MB. So 9.3GB memory usage by NumPy seems... awfully large? Not to mention 300s is a rather long time to take when printing.
(This was originally reported to JAX as https://github.com/google/jax/issues/7301; JAX simply calls NumPy to print arrays.)
Error message:
n/a
NumPy/Python version information:
In [1]: import sys, numpy; print(numpy.__version__, sys.version)
1.21.1 3.7.2 (default, Jan 13 2021, 20:29:55)
[Clang 11.0.0 (clang-1100.0.33.17)]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided NumPy/Python reproduction with x = np.ones([2]*25) and str(x), then trace the array string-conversion path. Compare peak memory and runtime with the reported 9.3GB and 298 seconds; done means printing the same array no longer incurs the reported excessive resource usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100