Consider pipelining merge and fetch tasks during re-partitioning
Open
performance
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
We currently write intermediate results during a re-partition job to disk 3 times (once to a file on the source, once to a file on the destination, and once to a tuple store on the destination).
Once we implement a custom scan for intermediate results (see also #3402), we will only write it 2 times.
We can avoid the other write on the destination by writing the intermediate results to named pipes and concurrently starting the next set of tasks and reading from the named pipes in (an optimised version of) read_intermediate_results.
Contributor guide
Assessment
This issue has not been assessed yet.