lance-format / lance-format/lance

bug: Session::new eagerly creates a LocalSpillStore temp dir and panics when it cannot

Open
#8,948 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Description

Session::new builds a spill store eagerly, and its constructor panics rather than returning an error. rust/lance/src/session.rs:130 and :150 both do Arc::new(LocalSpillStore::default()), and the Default impl is Self::new().expect("failed to create temp directory for LocalSpillStore") (rust/lance-io/src/spill.rs:233). So every session creates a temp directory up front, whether or not anything ever spills, and a machine where that directory cannot be created takes down any code path that opens a dataset.

On Windows CI runners this happens intermittently, and it lands on tests that have nothing to do with spilling. From windows-build, in io::commit::conflict_resolver::tests::test_data_overlay_finish_conflicts_with_row_moving_update::case_2_coverage_disjoint_from_moved_row:

failed to create temp directory for LocalSpillStore: IO { source: Custom { kind: PermissionDenied,
error: PathError { path: "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\.tmp9uNyQx",
err: Os { code: 5, kind: PermissionDenied, message: "Access is denied." } } }
Why it matters

The panic comes out of a Default impl, so no caller can handle it: a library user gets a process abort where an error would do. It also fires for sessions that never spill, since nothing about Session::new knows whether a query will need one.

Expected behavior

Either create the spill store on first use, so a session that never spills never touches the temp directory, or keep the eager construction and propagate the failure as an error from the session constructor.

Environment

Windows runners in GitHub Actions, intermittently. Not observed on Linux or macOS.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read rust/lance/src/session.rs at lines 130 and 150 and rust/lance-io/src/spill.rs around line 233 to trace how Session::new obtains LocalSpillStore. Run the named Windows failure test and verify that sessions which never spill no longer panic on temp-directory creation, or that the failure is returned as an error, with regression coverage for the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.