lance-format / lance-format/lance
Memory pool failures for query::primitives::test_query_integer::case_7_uint32
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
CI failure log:
It looks like there are memory pool issues caused by external sorts. If this is happening easily in CI, it means users are likely to run into this easily as well. We shouldn't just tune the CI job; we should make sure we don't run into it with out-of-the-box settings.
It might mean something is wrong with batch size or the default pool is too small.
thread 'query::primitives::test_query_integer::case_7_uint32' (17107) panicked at rust/lance/tests/utils/mod.rs:247:17:
Failed to create index on column 'value' with type BTree: LanceError(IO): Resources exhausted: Additional allocation failed for ExternalSorterMerge[0] with top memory consumers (across reservations) as:
ExternalSorterMerge[0]#15370(can spill: false) consumed 40.0 MB, peak 40.0 MB,
ExternalSorterMerge[0]#15384(can spill: false) consumed 40.0 MB, peak 40.0 MB,
ExternalSorterMerge[0]#15358(can spill: false) consumed 40.0 MB, peak 40.0 MB,
ExternalSorter[0]#15357(can spill: true) consumed 1516.0 B, peak 1516.0 B,
ExternalSorter[0]#15369(can spill: true) consumed 1331.0 B, peak 1331.0 B,
ExternalSorter[0]#15383(can spill: true) consumed 18.0 B, peak 18.0 B,
ExternalSorter[0]#15393(can spill: true) consumed 0.0 B, peak 0.0 B,
ExternalSorterMerge[0]#15394(can spill: false) consumed 0.0 B, peak 0.0 B.
Error: Failed to allocate additional 40.0 MB for ExternalSorterMerge[0] with 0.0 B already allocated for this reservation - 30.0 MB remain available for the total memory pool: fair(pool_size: 150.0 MB), rust/lance-datafusion/src/chunker.rs:49:46
https://github.com/lance-format/lance/actions/runs/35275325532/job/105384441938?pr=9358
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with query::primitives::test_query_integer::case_7_uint32 and the failure location in rust/lance/tests/utils/mod.rs:247. Read rust/lance-datafusion/src/chunker.rs:49 and trace the external-sort memory-pool allocation path. Done means the index creation no longer exhausts memory under the out-of-the-box settings, with the regression covered by the failing test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100