citusdata / citusdata/citus

INSERT .. SELECT should rely more on recursive planning

Open
#2,160 2 comments 0 reactions 0 assignees View on GitHub
feature performance
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

This idea brought up by @marcocitus. I'm opening the issue to not lose track of it since this might be a very useful improvement for some of the users.

Citus has a binary decision for `INSERT .. SELECT` queries: Push down the whole query vs Pull all the results to coordinator and push it back.

Instead, with recursive planning and subquery pushdown, we've enabled a more sophisticated way of executing `SELECT` queries such that only a small portion of the query could be recursively planned (e.g., pull - push back), and the rest of the query can be pushed down to the workers only that part being recursively planned.

An example:
```SQL
SELECT user_id
FROM users_table
WHERE user_id IN (SELECT user_id FROM users_table WHERE value_1 >= 1 AND value_1 <= 2)
AND user_id IN (SELECT user_id FROM users_table WHERE value_1 >= 3 AND value_1 <= 4)
AND value_2 IN (SELECT user_id FROM users_table WHERE value_1 >= 5 AND value_1 <= 6) ORDER BY 1 DESC LIMIT 3;

DEBUG: generating subplan 9_1 for subquery SELECT user_id FROM public.users_table WHERE ((value_1 OPERATOR(pg_catalog.>=) 5) AND (value_1 OPERATOR(pg_catalog.<=) 6))
DEBUG: Plan 9 query after replacing subqueries and CTEs: SELECT user_id FROM public.users_table WHERE ((user_id OPERATOR(pg_catalog.=) ANY (SELECT users_table_1.user_id FROM public.users_table users_table_1 WHERE ((users_table_1.value_1 OPERATOR(pg_catalog.>=) 1) AND (users_table_1.value_1 OPERATOR(pg_catalog.<=) 2)))) AND (user_id OPERATOR(pg_catalog.=) ANY (SELECT users_table_1.user_id FROM public.users_table users_table_1 WHERE ((users_table_1.value_1 OPERATOR(pg_catalog.>=) 3) AND (users_table_1.value_1 OPERATOR(pg_catalog.<=) 4)))) AND (value_2 OPERATOR(pg_catalog.=) ANY (SELECT intermediate_result.user_id FROM read_intermediate_result('9_1'::text, 'binary'::citus_copy_format) intermediate_result(user_id integer)))) ORDER BY user_id DESC LIMIT 3
DEBUG: push down of limit count: 3
```

As the output shows, we've only pull-push one of the subqueries. However, if we run the same query inside an `INSERT ... SELECT`, we'd have to pull-push all the subquery once.

```SQL
INSERT INTO events_table(user_id)
SELECT user_id
FROM users_table
WHERE user_id IN (SELECT user_id FROM users_table WHERE value_1 >= 1 AND value_1 <= 2)
AND user_id IN (SELECT user_id FROM users_table WHERE value_1 >= 3 AND value_1 <= 4)
AND value_2 IN (SELECT user_id FROM users_table WHERE value_1 >= 5 AND value_1 <= 6) ORDER BY 1 DESC LIMIT 3;
DEBUG: LIMIT clauses are not allowed in distributed INSERT ... SELECT queries
DEBUG: Collecting INSERT ... SELECT results on coordinator
DEBUG: generating subplan 12_1 for subquery SELECT user_id FROM public.users_table WHERE ((value_1 OPERATOR(pg_catalog.>=) 5) AND (value_1 OPERATOR(pg_catalog.<=) 6))
DEBUG: Plan 12 query after replacing subqueries and CTEs: SELECT user_id FROM public.users_table WHERE ((user_id OPERATOR(pg_catalog.=) ANY (SELECT users_table_1.user_id FROM public.users_table users_table_1 WHERE ((users_table_1.value_1 OPERATOR(pg_catalog.>=) 1) AND (users_table_1.value_1 OPERATOR(pg_catalog.<=) 2)))) AND (user_id OPERATOR(pg_catalog.=) ANY (SELECT users_table_1.user_id FROM public.users_table users_table_1 WHERE ((users_table_1.value_1 OPERATOR(pg_catalog.>=) 3) AND (users_table_1.value_1 OPERATOR(pg_catalog.<=) 4)))) AND (value_2 OPERATOR(pg_catalog.=) ANY (SELECT intermediate_result.user_id FROM read_intermediate_result('12_1'::text, 'binary'::citus_copy_format) intermediate_result(user_id integer)))) ORDER BY user_id DESC LIMIT 3
DEBUG: push down of limit count: 3
```

Some additional benefits would be users will be able to use more `ON CONFLICT` and `RETURNING` with `INSERT...SELECT`s given that we'd be able to push down a lot more `INSERT .. SELECT` queries by recursively planning a smaller part of the `SELECT`

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by tracing the planner paths for distributed INSERT ... SELECT, recursive planning, and subquery pushdown, using the SQL examples as behavioral cases. Done means supported INSERT ... SELECT queries can recursively plan only the necessary subqueries instead of collecting all results on the coordinator.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql, sql
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.