Copy directly into table in worker_append_table_to_shard
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
`worker_append_table_to_shard` currently copies data twice, once from the table on the source node into a file, and once from the file into the target table. We could speed up rebalancing and other operations by copying directly into the target table as rows from the source table are coming in.
For simplicity (and transactional correctness), it may be preferable to use a function on the source node that "pushes" the data to the target node, instead `worker_append_table_to_shard`, which pulls the data from the source node to the target node. The push function could for example be implemented by running an SPI query and sending the rows over the COPY protocol, whereas a pull function would have to parse the incoming lines and use low-level functions to insert them into the heap.
Contributor guide
Assessment
This issue has not been assessed yet.