lance-format / lance-format/lance

Fast way to verify and eliminate corrupted data files

Open
#6,694 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

In the actual production process, issues with dirty files may occur (such as unexpected commits during distributed writes), along with file corruption caused by storage-layer problems. We need a fast and convenient way to identify dirty files and even remove them from the Manifest to quickly restore the production availability of Lance tables.

For instance, occasional issues may occur in online environments

External error: Wrapped error: LanceError(IO): Object at location default.db/xxx.lance/data/0010001011101101010000109245a64c729acf551b4a3bc022.lance not found: 
Error performing GET https://xxxx/default.db/xxxx.lance/data/0010001011101101010000109245a64c729acf551b4a3bc022.lance in 2.891921ms 
Server returned non-2xx status code: 404 Not Found: <?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>
<Error><Code>NoSuchKey</Code><RequestIdxxxx</RequestId><HostId>xxxx</HostId><Message>The specified key does not exist.</Message><EC>xxxxx</EC>

<Key>default.db/xxxx.lance/data/0010001011101101010000109245a64c729acf551b4a3bc022.lance</Key></Error>, 
/rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/core/src/ops/[function.rs:250](http://function.rs:250/):5, /home/runner/work/lance/lance/rust/lance-io/src/[scheduler.rs:395](http://scheduler.rs:395/):17

We can quickly eliminate dirty files through the capabilities provided by the current PR.

./target/debug/lance-tools \
dataset repair-manifest \
-s "$LANCE_TOS_DATASET_ROOT" \
--storage-options "$LANCE_STORAGE_OPTIONS_JSON" \
--remove-data-files "FILE1,FILE2"

It also provides data file verification, as well as the capabilities of quickly locating files and performing rollbacks.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the lance-tools dataset repair-manifest entry point and its --remove-data-files command shown in the issue. Verify how the tool performs data-file verification, locates files, removes them from the Manifest, and supports rollbacks; done means these recovery operations work for corrupted or missing files.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, databases, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.