lance-format / lance-format/lance
Column Statistic improvment list after MVP
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
As we are wrapping up on the dataset level column statistics MVP, there are some high priority todo items:
[ ] Support column stats generation for partial dataset. Currently the dataset will only be generated when all the files in the dataset has column stats written. A naive implementation is just to build the stats from the old fragment file rather than terminate early.
[ ] Expose a Python API from the rust implementation.
[ ] Exposing more stats in the discussion thread #4540
More could be found in the design doc
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the dataset-level column statistics MVP in pull request #5639 and the design document linked in the issue. Review discussion #4540 for the additional statistics requirements. Done means addressing the listed partial-dataset generation, Python API exposure, and expanded-statistics items.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100