lance-format / lance-format/lance
feature: let the v3 shufflers own their shuffle scratch directory
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Problem
The v3 shufflers (IvfShuffler, TwoFileShuffler) hold only output_dir: Path, so whoever creates the directory has to keep a TempStdDir guard alive for the whole build. #8993 fixed one instance of that going wrong: prepare_vector_segment_build dropped the guard when it returned, the shuffle writer re-created the directory through create_dir_all(parent), and from then on nothing owned it, so every vector index build leaked a full shuffled copy of the vector column.
That fix works, but the invariant it rests on is a naming convention rather than anything the compiler checks. let (.., _) = prepare_vector_segment_build(..) compiles, binds nothing, drops the guard at the end of the statement, and the leak is back. #[must_use] does not help, because binding to _ counts as a use. Three call sites across rust/lance/src/index/vector.rs and rust/lance/src/index/vector/ivf.rs each keep their own guard alive by hand for the same reason, and the regression test only reaches one of them.
The shape this should have
The legacy shuffler already does it the other way round: IvfShuffler in rust/lance-index/src/vector/ivf/shuffler.rs carries owned_temp_dir: Option<Arc<tempfile::TempDir>> and cleans up when the last holder drops, with try_new taking Option<Path> so a caller either supplies a directory it owns or asks the shuffler to make one.
Giving the v3 shufflers the same option lets the three build paths drop their manual guards, and covers the distributed build path that the current regression test does not reach.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the v3 IvfShuffler and TwoFileShuffler definitions and the three call sites in rust/lance/src/index/vector.rs and rust/lance/src/index/vector/ivf.rs; compare their ownership flow with the legacy IvfShuffler in rust/lance-index/src/vector/ivf/shuffler.rs. Trace the existing regression test and distributed build path; done means all three paths no longer require manual TempStdDir guards and scratch data is cleaned up by shuffler ownership.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100