lance-format / lance-format/lance
bug: spawn multiple encoding tasks inside one batch number of rows of one column
Open
Nobody has claimed this yet.
A-encoding
bug
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently, we spawn only one encoding task per batch number rows in a single column, which can be problematic for datasets like MS MARCO where the third column is significant larger than the other two(more than 100x).
larger column needs more encoding tasks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the code that schedules encoding tasks for a batch and inspect how task counts are derived across columns. Reproduce the imbalance with a dataset whose third column is much larger, then verify that the larger column receives multiple encoding tasks within the batch.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100