lance-format / lance-format/lance
Epic: memory management
Open
Nobody has claimed this yet.
epic
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
In lots of places, it is possible to OOM. We should
Issues
- Allow setting (index, metadata) cache size in bytes
- Compaction memory limit https://github.com/lancedb/lancedb/issues/2325
- Memory limit in merge insert
- Figure out available memory from system: github.com/valpackett/systemstat
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This is an epic covering four separate memory-management concerns: cache sizing, compaction limits, merge-insert limits, and system-memory detection. No files or tests are named; start by selecting one checklist item and tracing its existing implementation. Done means the selected operation enforces the intended memory constraint and has coverage for it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100