lance-format / lance-format/lance

feature: let the v3 shufflers own their shuffle scratch directory

Open
#9,264 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Problem

The v3 shufflers (IvfShuffler, TwoFileShuffler) hold only output_dir: Path, so whoever creates the directory has to keep a TempStdDir guard alive for the whole build. #8993 fixed one instance of that going wrong: prepare_vector_segment_build dropped the guard when it returned, the shuffle writer re-created the directory through create_dir_all(parent), and from then on nothing owned it, so every vector index build leaked a full shuffled copy of the vector column.

That fix works, but the invariant it rests on is a naming convention rather than anything the compiler checks. let (.., _) = prepare_vector_segment_build(..) compiles, binds nothing, drops the guard at the end of the statement, and the leak is back. #[must_use] does not help, because binding to _ counts as a use. Three call sites across rust/lance/src/index/vector.rs and rust/lance/src/index/vector/ivf.rs each keep their own guard alive by hand for the same reason, and the regression test only reaches one of them.

The shape this should have

The legacy shuffler already does it the other way round: IvfShuffler in rust/lance-index/src/vector/ivf/shuffler.rs carries owned_temp_dir: Option<Arc<tempfile::TempDir>> and cleans up when the last holder drops, with try_new taking Option<Path> so a caller either supplies a directory it owns or asks the shuffler to make one.

Giving the v3 shufflers the same option lets the three build paths drop their manual guards, and covers the distributed build path that the current regression test does not reach.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the v3 IvfShuffler and TwoFileShuffler definitions and the three call sites in rust/lance/src/index/vector.rs and rust/lance/src/index/vector/ivf.rs; compare their ownership flow with the legacy IvfShuffler in rust/lance-index/src/vector/ivf/shuffler.rs. Trace the existing regression test and distributed build path; done means all three paths no longer require manual TempStdDir guards and scratch data is cleaned up by shuffler ownership.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.