citusdata / citusdata/citus

Problem with node activation after disabling node temporarily when having partitioned tables

Open
#6,701 2 comments 0 reactions 1 assignee Claimed by @emelsimsek View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Hi,
We're using Citus 11.1.4

## Steps to reproduce

1. Create distributed table with partitions
```sql
CREATE TABLE t1 (
item_id text NOT NULL,
date_in timestamp NOT NULL,
PRIMARY KEY (item_id, date_in)
) PARTITION BY RANGE (date_in);

CREATE TABLE t1_2022_jan PARTITION OF t1
FOR VALUES FROM ('2022-01-01') TO ('2022-02-01');

CREATE TABLE t1_2022_feb PARTITION OF t1
FOR VALUES FROM ('2022-02-01') TO ('2022-03-01');

SELECT create_distributed_table('t1', 'item_id');
```

2. Move shards off the designated node and disable the node temporarily
```sql
SELECT * FROM citus_drain_node('citus_worker_3', 5432);
SELECT * FROM citus_disable_node('citus_worker_3', 5432);
```
3. Add a new partition to the distributed table
```sql
CREATE TABLE t1_2022_mar PARTITION OF t1
FOR VALUES FROM ('2022-03-01') TO ('2022-04-01');
```
4. Activate node
```sql
SELECT * FROM citus_activate_node('citus_worker_3', 5432);
```
getting error:
```bash
WARNING: relation "public.t1_2022_mar" does not exist
ERROR: failure on connection marked as essential: citus_worker_3:5432
```

In the nodes logs
```bash
ERROR: relation "public.t1_2022_mar" does not exist
2023-02-06 13:17:33.051 UTC [714] STATEMENT: SET citus.enable_ddl_propagation TO 'off';ALTER TABLE IF EXISTS public.t1 DETACH PARTITION public.t1_2022_jan;;ALTER TABLE IF EXISTS public.t1 DETACH PARTITION public.t1_2022_feb;;ALTER TABLE IF EXISTS public.t1 DETACH PARTITION public.t1_2022_mar;;SET citus.enable_ddl_propagation TO 'on';SELECT pg_catalog.worker_drop_sequence_dependency(logicalrelid::regclass::text) FROM pg_dist_partition;SELECT worker_drop_shell_table(logicalrelid::regclass::text) FROM pg_dist_partition;SET citus.enable_ddl_propagation TO 'off';SELECT worker_create_or_alter_role('postgres', 'CREATE ROLE postgres SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION BYPASSRLS CONNECTION LIMIT -1 PASSWORD NULL VALID UNTIL ''infinity''', 'ALTER ROLE postgres SUPERUSER CREATEDB CREATEROLE INHERIT LOGIN REPLICATION BYPASSRLS CONNECTION LIMIT -1 PASSWORD NULL VALID UNTIL ''infinity''');ALTER DATABASE docker OWNER TO postgres;;CREATE SCHEMA IF NOT EXISTS public AUTHORIZATION pg_database_owner;SET ROLE pg_database_owner;GRANT USAGE ON SCHEMA public TO pg_database_owner;;GRANT CREATE ON SCHEMA public TO pg_database_owner;;RESET ROLE;SET ROLE pg_database_owner;GRANT USAGE ON SCHEMA public TO PUBLIC;;RESET ROLE;CREATE TABLE public.t1 (item_id text NOT NULL, date_in timestamp without time zone NOT NULL) PARTITION BY RANGE (date_in) ;ALTER TABLE public.t1 OWNER TO postgres;SELECT worker_create_truncate_trigger('public.t1');ALTER TABLE public.t1 ADD CONSTRAINT t1_pkey PRIMARY KEY (item_id, date_in);CREATE TABLE public.t1_2022_jan (item_id text NOT NULL, date_in timestamp without time zone NOT NULL) USING heap;ALTER TABLE public.t1_2022_jan OWNER TO postgres;SELECT worker_create_truncate_trigger('public.t1_2022_jan');ALTER TABLE public.t1_2022_jan ADD CONSTRAINT t1_2022_jan_pkey PRIMARY KEY (item_id, date_in);CREATE TABLE public.t1_2022_feb (item_id text NOT NULL, date_in timestamp without time zone NOT NULL) USING heap;ALTER TABLE public.t1_2022_feb OWNER TO postgres;SELECT worker_create_truncate_trigger('public.t1_2022_feb');ALTER TABLE public.t1_2022_feb ADD CONSTRAINT t1_2022_feb_pkey PRIMARY KEY (item_id, date_in);CREATE TABLE public.t1_2022_mar (item_id text NOT NULL, date_in timestamp without time zone NOT NULL) USING heap;ALTER TABLE public.t1_2022_mar OWNER TO postgres;SELECT worker_create_truncate_trigger('public.t1_2022_mar');ALTER TABLE public.t1_2022_mar ADD CONSTRAINT t1_2022_mar_pkey PRIMARY KEY (item_id, date_in);SET citus.enable_ddl_propagation TO 'on';SET citus.enable_ddl_propagation TO 'off';ALTER TABLE public.t1 ATTACH PARTITION public.t1_2022_jan FOR VALUES FROM ('2022-01-01 00:00:00') TO ('2022-02-01 00:00:00');;ALTER TABLE public.t1 ATTACH PARTITION public.t1_2022_feb FOR VALUES FROM ('2022-02-01 00:00:00') TO ('2022-03-01 00:00:00');;ALTER TABLE public.t1 ATTACH PARTITION public.t1_2022_mar FOR VALUES FROM ('2022-03-01 00:00:00') TO ('2022-04-01 00:00:00');;SET citus.enable_ddl_propagation TO 'on'
```
## Workaround
1. Re-create database(s) on the disabled node.
```sql
DROP DATABASE IF EXISTS db1;
CREATE DATABASE db1;
\c db1
CREATE EXTENSION citus;
```
2. Activate node on the re-created database(s).
```
SELECT * from citus_activate_node('citus_worker_3', 5432);
```
(Above steps has been reproduced in the docker environment using https://github.com/citusdata/docker/blob/master/docker-compose.yml)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.