lance-format / lance-format/lance
.lance files can not be symlinks: "file size is too small"
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Hi.
Based on my original issue in LanceDB: https://github.com/lancedb/lancedb/issues/3197
If I download a LanceDB dataset from HuggingFace like that:
hf download --type dataset username/repo
Then in the snapshots/ subfolder of the downloaded artifacts in HuggingFace cache there will be symlinks to the actual blobs:
10010110010001001001110179af84444b9f7d4ff87ee2253b.lance@ -> ../../../../../blobs/be64f20e5723bd0a27cfdbdb41cf7d6fad94cd572a71973b717fb8340f4310c5
That's how HuggingFace stores files. But LanceDB fails to open a table with such symlinks:
error: lance error: LanceError(IO): file size is too small, /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-io-3.0.1/src/utils.rs:88:20 Caused by: LanceError(IO): file size is too small, /Users/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/lance-io-3.0.1/src/utils.rs:88:20 Caused by: file size is too small
code: "GenericFailure"
Like it's literally trying to open and read a .lance symlink file as is without following it to the original source. The error originates here: https://github.com/lance-format/lance/blob/dabfb933c789b661805c36c9397d4045a1bc76ef/rust/lance-io/src/utils.rs#L86-L89
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with a HuggingFace dataset containing symlinked .lance files, then inspect rust/lance-io/src/utils.rs around lines 86-89 where the reported error originates. Trace how the file is opened and verify that the dataset can be opened through the cached symlink, with regression coverage for this case if the surrounding tests provide an appropriate place.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100