Add a shared memory connection pool to the adaptive executor
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
There's certain scenarios where our current executor design doesn't work well. The main scenario is multi shard queries, where each query to a shard only takes very little time (i.e. less than 1ms). This causes latency between nodes to become the bottleneck. By default each backend caches one open connection to each worker. If such a query needs to access the same worker multiple times, there's a problem with two suboptimal solutions:
1. Either more connections need to be opened, because of TLS this will result in multiple roundtrips.
2. The accesses are done sequentially, again requiring multiple roundtrips, because the first query needs to wait for the second one.
One idea to solve this is to add a connection pool in shared memory. This can be added to the adaptive executor, without requiring a whole redesign. Instead of actually "opening" a new connection, a backend would take a connection from the shared memory pool if it exists. If it doesn't exist it would actually open a new connection. When closing a connection, the connection would be put back into the shared memory pool.
@onderkalaci said this:
> Few thing we should be careful about:
> 1. Today we have only shared integer counters which does simple calculations, and with high concurrency, the locks to access the shared mem. becomes a bottleneck. So, should be very careful not to make the problem even bigger
> 2. citus.max_cached_connection_lifetime should still work
Contributor guide
Assessment
This issue has not been assessed yet.