Overcome some transactional limitations when reference tables involved in a distributed transaction
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
With #2798, we've eliminated almost all of the distributed transactional limitations, leaving few cases with reference tables.
These cases are already covered in our regression tests, but opening issue for keeping track of it. The workaround is to `set citus.max_adaptive_executor_pool_size TO 1`, which prevents parallelism, so yields slow executions.
Setup for the tests:
```SQL
-- 3 worker nodes in the cluster
SET citus.shard_count TO 4;
SET citus.next_shard_id TO 1;
CREATE TABLE table_1 (key int, value text);
SELECT create_distributed_table('table_1', 'key');
CREATE TABLE ref_table (key int, value text);
SELECT create_reference_table('ref_table');
```
case 1: Modify a reference table over `connection 1`, later modify a distributed table over `connection 1` and `connection 2` in parallel. Finally, joining these two tables would fail because the executor cannot choose the accurate connection. When it need to execute the join between table_1_2 and ref_table_1, it cannot choose any connections. If it chooses connection 1, the modification to table_1_2 would not be visible. If it chooses the connection 2, the modification to ref_table_1 would not be visible. Instead, the executor bails out with an error:
```SQL
BEGIN;
-- make a modification over connection 1 to worker 1
INSERT INTO ref_table VALUES (1,'istanbul');
-- copy over connections 1 to table_1_1 and connection 2 to table_1_2 on worker 1
\COPY table_1 FROM STDIN WITH CSV
10, 'istanbul'
11, 'istanbul'
\.
-- ref_table is modified over one connection 1
-- table_1 is modified over two connections connection 1 and connection 2
-- so the executor is undecided, it cannot pick either connection 1 or connection 2
-- because table_1_2 is modified over connection 2
-- but ref_table_1 is modified over connection 1
-- so choosing either connection 1 or connection 2 would yield wrong
-- results
SELECT count(*) FROM table_1 JOIN ref_table USING(key);
ERROR: cannot perform query with placements that were modified over multiple connections
END;
```
case 2: Modify a distributed table over `connection 1` and `connection 2` in parallel. Join reference table with the distributed table multiple times. The first one uses `connection 1`, the second one user `connection 2` as those connections modified the placements that the join hits earlier in the transaction.
Finally, a DDL on the reference table would error out. The reason is that, choosing `connection 1` would deadlock with `connection 2` and vice versa as those connections hold `AccessShareLock` because of the previous reads. And, DDL would acquire `AccessExclusiveLock`, which would end-up a self-distributed-deadlock.
```SQL
BEGIN;
-- copy over connections 1 to table_1_1 and connection 2 to table_1_2 on worker 1
\COPY table_1 FROM STDIN WITH CSV
10, 'istanbul'
11, 'istanbul'
\.
-- read from the reference table over connection_1
SELECT count(*) FROM table_1 JOIN ref_table USING (key) WHERE key = 10;
-- read from the reference table over connection_2
SELECT count(*) FROM table_1 JOIN ref_table USING (key) WHERE key = 11;
-- perform a DDL command on a reference table
-- but since the reference table is accessed over multiple connections
-- in the previous commands, the executor is undecided
-- it cannot choose any connection to execute the DDL command
-- since it'd deadlock with the other one
ALTER TABLE ref_table ADD COLUMN last_update timestamptz;
ERROR: cannot perform DDL on placement 524, which has been read over multiple connections
END;
```
I cannot come-up with solutions easily to this situation.
Contributor guide
Assessment
This issue has not been assessed yet.