Shard splitting to the same node can make node run out of disk space due to lots of unconsumed
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
When doing a non blocking split to the same node, and the shard group is large (e.g. 1TB) it will copy all this data to the local node and thus create just as much WAL. This WAL can then not be removed by Postgres, because the replication slot is not being consumed yet during the COPY. I'm not sure if there's much we can do here. But we should probably document this, and if/when we include shardsplits into the rebalancer we should take this kind of thing into account.
Something similar was happening for this user when splitting to the coordinator. I'm not super sure what the cause of that is, maybe WAL could not be removed on the coordinator because the transaction that was doing the shard split was still open.
Reported on slack (long thread): https://citus-public.slack.com/archives/C0XRHT1KJ/p1691573609184889
Contributor guide
Assessment
This issue has not been assessed yet.