citusdata / citusdata/citus

Time partitions seemingly breaking themselves at random on distributed tables.

Open
#6,809 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We've been working with citus distributed and partitionned tables for a few weeks now, and some partitions broke at random on distributed tables. This makes the table unable to run a query without time constraints, or just on the partition :
![image](https://user-images.githubusercontent.com/31518175/228763455-09a8dbdb-504b-478f-bb00-05ef75b957f0.png)

This does not happen for the neighboring partitions :
![image](https://user-images.githubusercontent.com/31518175/228763555-a3d2665c-9aa4-4f4f-b04a-be7778ebaa29.png)

And only happens on one of our worker nodes (w1) :
![image](https://user-images.githubusercontent.com/31518175/228766410-7bf4a4c0-c808-4057-b0a0-65811976df48.png)

Apparently, the partition randomly lost 8KB of data according to citus_tables and citus_total_relation_size :
![image](https://user-images.githubusercontent.com/31518175/228763806-61867f14-48d0-414a-9999-ca3540a4468b.png)

![image](https://user-images.githubusercontent.com/31518175/228763834-00bf2a99-f887-4b15-ad01-5d7cb8aa06c8.png)

This already happened three times (as we know of) on two distinct tables.
The broken partitions are January 2025 and November 2027 on the first table and March 2027 on the second table.
We know that the partition on the second table broke on March 19th of this year.

We are using the Azure Cosmos DB for PostgreSQL Cluster to host the database.

Any help to recover/repair these partitions are greatly appreciated because our current solution is to drop the partition and regenerate it with create_time_partitions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.