citusdata / citusdata/citus

Get rid of unnecessary calls to RouterSelectTask() for INSERT ... SELECT

Open
#1,245 1 comment 0 reactions 0 assignees View on GitHub
technical debt
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

In the current implementation of `INSERT ... SELECT` queries, the following algorithm is followed:

```python

for targetShardInterval in shardIntervalsOfInsertedTable:

InstantiateQualToTargetShardInterval()
call RouterSelectTask()
call ShardPruning() on all shards
```

The above algorithm does not scale for high-shard counts. We should probably implement this more efficient.

One possible approach could be (as prototyped in #1240 ):

```python

prunedShardList = PruneShardList()

for targetShardInterval in shardIntervalsOfInsertedTable:

if targetShardInterval not in prunedShardList:
continue;

InstantiateQualToTargetShardInterval()

call RouterSelectTask() with input shard list as prunedShardList (instead of the whole shards)
call ShardPruning() with input shard list as prunedShardList (instead of the whole shards)
```

This could dramatically decrease the planning time with high shard counts where the query hits only few shards.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.