lance-format / lance-format/lance

.lance files can not be symlinks: "file size is too small"

Open
#6,497 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Hi.

Based on my original issue in LanceDB: https://github.com/lancedb/lancedb/issues/3197

If I download a LanceDB dataset from HuggingFace like that:

hf download --type dataset username/repo

Then in the snapshots/ subfolder of the downloaded artifacts in HuggingFace cache there will be symlinks to the actual blobs:

10010110010001001001110179af84444b9f7d4ff87ee2253b.lance@ -> ../../../../../blobs/be64f20e5723bd0a27cfdbdb41cf7d6fad94cd572a71973b717fb8340f4310c5

That's how HuggingFace stores files. But LanceDB fails to open a table with such symlinks:

error: lance error: LanceError(IO): file size is too small, /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-io-3.0.1/src/utils.rs:88:20  Caused by: LanceError(IO): file size is too small, /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-io-3.0.1/src/utils.rs:88:20    Caused by: file size is too small
 code: "GenericFailure"

Like it's literally trying to open and read a .lance symlink file as is without following it to the original source. The error originates here: https://github.com/lance-format/lance/blob/dabfb933c789b661805c36c9397d4045a1bc76ef/rust/lance-io/src/utils.rs#L86-L89

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with a HuggingFace dataset containing symlinked .lance files, then inspect rust/lance-io/src/utils.rs around lines 86-89 where the reported error originates. Trace how the file is opened and verify that the dataset can be opened through the cached symlink, with regression coverage for this case if the surrounding tests provide an appropriate place.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.