Parallelize `infer_schema`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Currently `infer_schema` runs in a single thread.
Also it uses `buffered` instead of `buffer_unordered` (the latter will be better if some reads are taking much longer than others).
You can see below it takes ~130ms just to load the schema/metadata for the (100) clickbench_partitioned files.
### Describe the solution you'd like
We should be able to use all threads.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Locate infer_schema and inspect how it currently performs reads with buffered. Compare the behavior with buffer_unordered, then use the clickbench_partitioned files as the performance case. Done means schema and metadata reads can use all available threads and no longer wait in ordered batches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100