lance-format / lance-format/lance

Memory pool failures for query::primitives::test_query_integer::case_7_uint32

Open
#9,402 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

CI failure log:

It looks like there are memory pool issues caused by external sorts. If this is happening easily in CI, it means users are likely to run into this easily as well. We shouldn't just tune the CI job; we should make sure we don't run into it with out-of-the-box settings.

It might mean something is wrong with batch size or the default pool is too small.

thread 'query::primitives::test_query_integer::case_7_uint32' (17107) panicked at rust/lance/tests/utils/mod.rs:247:17:
Failed to create index on column 'value' with type BTree: LanceError(IO): Resources exhausted: Additional allocation failed for ExternalSorterMerge[0] with top memory consumers (across reservations) as:
  ExternalSorterMerge[0]#15370(can spill: false) consumed 40.0 MB, peak 40.0 MB,
  ExternalSorterMerge[0]#15384(can spill: false) consumed 40.0 MB, peak 40.0 MB,
  ExternalSorterMerge[0]#15358(can spill: false) consumed 40.0 MB, peak 40.0 MB,
  ExternalSorter[0]#15357(can spill: true) consumed 1516.0 B, peak 1516.0 B,
  ExternalSorter[0]#15369(can spill: true) consumed 1331.0 B, peak 1331.0 B,
  ExternalSorter[0]#15383(can spill: true) consumed 18.0 B, peak 18.0 B,
  ExternalSorter[0]#15393(can spill: true) consumed 0.0 B, peak 0.0 B,
  ExternalSorterMerge[0]#15394(can spill: false) consumed 0.0 B, peak 0.0 B.
Error: Failed to allocate additional 40.0 MB for ExternalSorterMerge[0] with 0.0 B already allocated for this reservation - 30.0 MB remain available for the total memory pool: fair(pool_size: 150.0 MB), rust/lance-datafusion/src/chunker.rs:49:46

https://github.com/lance-format/lance/actions/runs/35275325532/job/105384441938?pr=9358

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with query::primitives::test_query_integer::case_7_uint32 and the failure location in rust/lance/tests/utils/mod.rs:247. Read rust/lance-datafusion/src/chunker.rs:49 and trace the external-sort memory-pool allocation path. Done means the index creation no longer exhausts memory under the out-of-the-box settings, with the regression covered by the failing test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.