citusdata / citusdata/citus

Consider not claiming intermediate result connections exclusively

Open
#3,312 2 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

When there are multiple levels of recursive planning (hence broadcasting intermediate results), Citus claims one connection per level exclusively via `ClaimConnectionExclusively()` in `RemoteFileDestReceiverStartup()`. This means that, if there are X levels of planning, Citus would use X connections per node.

Instead, we could optimize the overall execution to use 1 connection per node. The following is a quote from the chat with @marcocitus

```

I think RemoteFileDestReceiverShutdown only happens once the executor is done with the outer query
​ and the entire inner query execution happens within the execution of the outer query
​ technically, I think we might be able to defer opening connections from RemoteFileDestReceiverStartup to RemoteFileDestReceiverReceive
​ so we rather wait with broadcasting the outer query until we have results from the outer query
​ since the inner query finishes execution before the outer query starts, that should work
​ that could be a useful optimisation for nested recursive plans

```

An example query for nested ctes (`random()` is to prevent CTE inlining ):
```SQL
WITH cte_1 AS (WITH cte_2 AS (SELECT user_id, random() FROM users_table) SELECT user_id,random() FROM users_table JOIN cte_2 USING (user_id))

SELECT count(*) FROM cte_1 JOIN users_table USING (user_id);

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.