[EPIC] Improve NUMA-awareness
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
This is a list to check where DataFusion w.r.t. NUMA-awareness and where we can improve.
I will list to some other tickets.
- https://github.com/apache/datafusion/issues/21719
- [ ] Support morsel-splitting in Parquet, other datasources for better parallelisation / distribution / prefetching
- [x] Read parquet metadata to use `ParquetMetaDataPushDecoder`
- [ ] Support/use pinning threads to cores to minimize context switches / cache misses)
- [ ] Support reading into thread-local storage using async io / `io_uring` instead of using `spawn_blocking`
- [ ] Split/decode row groups into sizes that fit well in cache (e.g. ~RecordBatch)
- [ ] Better query execution pipelining / scheduling to minimize context switching / optimize data locality
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the linked issue 21719 and review the NUMA-awareness checklist in this issue. The listed areas include Parquet morsel-splitting, ParquetMetaDataPushDecoder, thread pinning, async I/O, cache-sized row groups, and query scheduling; the work is complete only when a specific improvement is selected, implemented, and its checklist item can be marked done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100