citusdata / citusdata/citus

Consider pipelining merge and fetch tasks during re-partitioning

Open
#3,533 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We currently write intermediate results during a re-partition job to disk 3 times (once to a file on the source, once to a file on the destination, and once to a tuple store on the destination).

Once we implement a custom scan for intermediate results (see also #3402), we will only write it 2 times.

We can avoid the other write on the destination by writing the intermediate results to named pipes and concurrently starting the next set of tasks and reading from the named pipes in (an optimised version of) read_intermediate_results.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.