Consider pipelining map and fetch tasks during re-partitioning
Open
performance
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
During a map task in a re-partition join we currently write the full result of each task to a set of files and then fetch them to the destination. We could improve performance by starting the fetch concurrently with the map.
One approach could be to write results to a set of named pipes and have the COPY commands that are issued by the fetch function on the destination read from all named pipes that are destined for that worker.
Combined with #3533 and this allows full pipelining of the re-partition join.
Contributor guide
Assessment
This issue has not been assessed yet.