citusdata / citusdata/citus

Shard splitting to the same node can make node run out of disk space due to lots of unconsumed

Open
#7,125 5 comments 0 reactions 0 assignees View on GitHub
rebalancer user_reported
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

When doing a non blocking split to the same node, and the shard group is large (e.g. 1TB) it will copy all this data to the local node and thus create just as much WAL. This WAL can then not be removed by Postgres, because the replication slot is not being consumed yet during the COPY. I'm not sure if there's much we can do here. But we should probably document this, and if/when we include shardsplits into the rebalancer we should take this kind of thing into account.

Something similar was happening for this user when splitting to the coordinator. I'm not super sure what the cause of that is, maybe WAL could not be removed on the coordinator because the transaction that was doing the shard split was still open.

Reported on slack (long thread): https://citus-public.slack.com/archives/C0XRHT1KJ/p1691573609184889

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.