lance-format / lance-format/lance
Fast way to verify and eliminate corrupted data files
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
In the actual production process, issues with dirty files may occur (such as unexpected commits during distributed writes), along with file corruption caused by storage-layer problems. We need a fast and convenient way to identify dirty files and even remove them from the Manifest to quickly restore the production availability of Lance tables.
For instance, occasional issues may occur in online environments
External error: Wrapped error: LanceError(IO): Object at location default.db/xxx.lance/data/0010001011101101010000109245a64c729acf551b4a3bc022.lance not found:
Error performing GET https://xxxx/default.db/xxxx.lance/data/0010001011101101010000109245a64c729acf551b4a3bc022.lance in 2.891921ms
Server returned non-2xx status code: 404 Not Found: <?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>
<Error><Code>NoSuchKey</Code><RequestIdxxxx</RequestId><HostId>xxxx</HostId><Message>The specified key does not exist.</Message><EC>xxxxx</EC>
<Key>default.db/xxxx.lance/data/0010001011101101010000109245a64c729acf551b4a3bc022.lance</Key></Error>,
/rustc/f8297e351a40c1439a467bbbb6879088047f50b3/library/core/src/ops/[function.rs:250](http://function.rs:250/):5, /home/runner/work/lance/lance/rust/lance-io/src/[scheduler.rs:395](http://scheduler.rs:395/):17
We can quickly eliminate dirty files through the capabilities provided by the current PR.
./target/debug/lance-tools \
dataset repair-manifest \
-s "$LANCE_TOS_DATASET_ROOT" \
--storage-options "$LANCE_STORAGE_OPTIONS_JSON" \
--remove-data-files "FILE1,FILE2"
It also provides data file verification, as well as the capabilities of quickly locating files and performing rollbacks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the lance-tools dataset repair-manifest entry point and its --remove-data-files command shown in the issue. Verify how the tool performs data-file verification, locates files, removes them from the Manifest, and supports rollbacks; done means these recovery operations work for corrupted or missing files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, databases, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100