lance-format / lance-format/lance
[Feature] lance support mosaicml streaming
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
like this issue https://github.com/mosaicml/streaming/issues/832
Lance can perform random reads very well, and the primary key can bring good shuffle ability. However, streaming can provide a better shuffle algorithm for training.
So I bring up the possibility of this combination again here and want to discuss the ideas of the community here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked MosaicML Streaming issue and this issue's discussion to understand the proposed Lance and streaming combination. The work is not yet scoped to files or tests; done would require an agreed integration design for random reads, primary-key-based access, and training-data shuffling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100