Out of memory error while pipeline worked previously
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.3k
- Forks
- 871
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 5
Description
I am trying to replicate an earlier result, and as part of this I apply UMAP to my data set with size (253695, 6716).
The verbose printouts show that after UMAP builds the RF Forest and finishes the NN search, it crashes as it is unable to locate (6.35) GB of memory. I used the exact same code 2 months prior, on the same data, with no hardware changes whatsoever and it always worked (ran it about 50 times).
As I am trying to replicate an earlier result I would really want to keep the size of my data. What could I try out here, besides upgrading my memory (32 GB)? I'm a bit puzzled by being faced with an OOM error ~2 months after last running the same code and not changing a thing.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or reproducible entry point is named. Start by reviewing the UMAP stages reported in the verbose output, where the RF forest is built and the nearest-neighbor search finishes, then compare the conditions under which the 253695 × 6716 dataset is processed. Done means identifying the cause of the memory increase or documenting a reproducible mitigation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100