lance-format / lance-format/lance
Leverage object store bulk copy to efficiently deep_clone dataset
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Lance format now support deep_clone dataset: https://github.com/lance-format/lance/pull/5250
But the underlying is using single file copy interface. As we know all cloud storage have batch copy, we should ideally leverage that, but that requires upstream support.
This issue is to track this progress
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked Lance deep_clone support in pull request #5250 and the issue’s description of single-file copying. Then investigate the upstream support needed for object-store batch copy. Done means dataset deep_clone uses bulk copy rather than the single-file copy interface across supported cloud storage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100