Run out of memory on a 1.6 m dataset
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
Hi, great work for UMAP. I got oom when I run UMAP on a 1.6 m point dataset with 300 dimensions for each point. UMAP eats all my 32G memory in 2 minutes. I enabled verbose and it shows building RP forest with 69 trees. I also run it with low_memory=True, init='random', and the same oom.
Is there any equation to pre-calculate how many memory the dataset gonna use for a 1.6m X 300 dataset?
Thanks a lot!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the out-of-memory run with 1.6 million points, 300 dimensions, and the reported UMAP settings. Inspect the RP forest construction and low_memory behavior using the repository's existing entry points; done would mean establishing the memory cause or producing a defined, tested change that prevents or clearly reports the exhaustion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100