lincc-frameworks / lincc-frameworks/hyrax
Make chroma-db less resource intensive
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 41
- Forks
- 7
- Avg merge
- 5d 1h
- Merged PRs (30d)
- 8
Description
chroma-db as it is currently implemented has the following issues when scaling up to millions of images:-
- runs end with `oom` unless the entire chroma-db can fit on RAM. (for reference, a chroma-db with a million vectors is about 100GB; so this quickly gets out of hand as you scale up to 10 Million). As @drewoldag was pointing out; this is probably due to the chroma-db being loaded into memory for the write operation to happen?
- runs where I set chroma-db to True take at least almost an order of magnitude longer on the scale of millions (for inference with 1 million images on two A-40 GPUs; runtime is 3 hours without chroma-db; and > 24 hours with chroma-db). I am assuming a lot of time is being spent on I/O as data is written to the chroma-db
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are identified; start by locating the ChromaDB write path and profiling runs with ChromaDB enabled at the million-image scale. Done means the run no longer ends with OOM and the reported ChromaDB runtime is substantially reduced, while preserving the existing results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100