citusdata / citusdata/citus

Shard splits might hang up with concurrent transactions

Open
#6,790 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Steps to repro:

schema
```SQL
SELECT 1 FROM master_set_node_property('localhost', 5432, 'shouldhaveshards', true);
CREATE TABLE sensors(
measureid integer PRIMARY KEY,
measure_cnt bigint);

SELECT create_distributed_table('sensors', 'measureid');
insert into sensors select i,0 from generate_series(0, 1000) i;
```

concurrent pgbench:
```
-- on all nodes
pgbench -h 10.192.0.6 -f /tmp/f.sql -c 8 -j 4 -T 1200 -P 3 &
pgbench -h 10.192.0.5 -f /tmp/f.sql -c 8 -j 4 -T 1200 -P 3 &
pgbench -h 10.192.0.4 -f /tmp/f.sql -c 8 -j 4 -T 1200 -P 3 &
```

f.sql
```
\set aid random(1, 1000)

update sensors set measure_cnt=measure_cnt+1 WHERE measureid=:aid;
```

```SQL
select citus_split_shard_by_split_points(
shardid,
/* pick the split points, in this case we take the middle of the hash range*/
array[(shardminvalue::int + (shardmaxvalue::int - shardminvalue::int) / 2)::text],
/* send first half to node 2, second half to node 3 */
array[2, 3],
shard_transfer_mode := 'auto')
from pg_dist_shard ORDER BY random() LIMIT 1;
\watch 0.1

```

one of the `citus_split_shard_by_split_points` stalls, and we have the following error on the logs repeatedly :
```
2023-03-22 13:23:35.345 UTC [94825] LOG: logical replication apply worker for subscription "citus_shard_split_subscription_10_24" has started
2023-03-22 13:23:35.353 UTC [94826] LOG: starting logical decoding for slot "citus_shard_split_slot_3_10_24"
2023-03-22 13:23:35.353 UTC [94826] DETAIL: Streaming transactions committing after 0/129F5BF0, reading WAL from 0/129F1790.
2023-03-22 13:23:35.353 UTC [94826] STATEMENT: START_REPLICATION SLOT "citus_shard_split_slot_3_10_24" LOGICAL 0/0 (proto_version '3', publication_names '"citus_shard_split_publication_3_10_24"', binary 'true')
2023-03-22 13:23:35.355 UTC [94826] LOG: logical decoding found consistent point at 0/129F1790
2023-03-22 13:23:35.355 UTC [94826] DETAIL: Logical decoding will begin using saved snapshot.
2023-03-22 13:23:35.355 UTC [94826] STATEMENT: START_REPLICATION SLOT "citus_shard_split_slot_3_10_24" LOGICAL 0/0 (proto_version '3', publication_names '"citus_shard_split_publication_3_10_24"', binary 'true')
2023-03-22 13:23:35.356 UTC [94825] ERROR: publisher did not send replica identity column expected by the logical replication target relation "public.sensors_102117"
2023-03-22 13:23:35.356 UTC [94825] CONTEXT: processing remote data for replication origin "pg_17653" during message type "UPDATE" for replication target relation "public.sensors_102117" in transaction 1906605, finished at 0/129F5C48
2023-03-22 13:23:35.357 UTC [94532] LOG: background worker "logical replication worker" (PID 94825) exited with exit code 1

```

```SQL
-- pg_dist_node, nodeid=3 is coordinator
table pg_dist_node;
nodeid | groupid | nodename | nodeport | noderack | hasmetadata | isactive | noderole | nodecluster | metadatasynced | shouldhaveshards
--------+---------+------------+----------+----------+-------------+----------+----------+-------------+----------------+------------------
1 | 1 | 10.192.0.5 | 5432 | default | t | t | primary | default | t | t
2 | 2 | 10.192.0.6 | 5432 | default | t | t | primary | default | t | t
3 | 0 | 10.192.0.4 | 5432 | default | t | t | primary | default | t | t
(3 rows)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.