Get rid of unnecessary calls to RouterSelectTask() for INSERT ... SELECT
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
In the current implementation of `INSERT ... SELECT` queries, the following algorithm is followed:
```python
for targetShardInterval in shardIntervalsOfInsertedTable:
InstantiateQualToTargetShardInterval()
call RouterSelectTask()
call ShardPruning() on all shards
```
The above algorithm does not scale for high-shard counts. We should probably implement this more efficient.
One possible approach could be (as prototyped in #1240 ):
```python
prunedShardList = PruneShardList()
for targetShardInterval in shardIntervalsOfInsertedTable:
if targetShardInterval not in prunedShardList:
continue;
InstantiateQualToTargetShardInterval()
call RouterSelectTask() with input shard list as prunedShardList (instead of the whole shards)
call ShardPruning() with input shard list as prunedShardList (instead of the whole shards)
```
This could dramatically decrease the planning time with high shard counts where the query hits only few shards.
Contributor guide
Assessment
This issue has not been assessed yet.