citusdata / citusdata/citus

citus_rebalance_start gets stuck after changing shouldhaveshards

Open
#6,681 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

After changing the shouldhaveshards of a node that was receiving shard moves, moves kept failing:

```
2023-01-30 23:22:37.296 CET [13401] LOG: found scheduled background tasks, starting new background task queue monitor
2023-01-30 23:22:37.296 CET [13401] CONTEXT: Citus maintenance daemon for database 5 user 10
2023-01-30 23:22:37.309 CET [15229] LOG: task jobid/taskid started: 6/79
2023-01-30 23:22:37.309 CET [15229] CONTEXT: Citus Background Task Queue Monitor: postgres
2023-01-30 23:22:37.317 CET [15230] ERROR: Moving shards to a node that shouldn't have a shard is not supported
2023-01-30 23:22:37.317 CET [15230] HINT: Allow shards on the target node via SELECT * FROM citus_set_node_property('localhost', 1502, 'shouldhaveshards', true);
2023-01-30 23:22:37.317 CET [15230] CONTEXT: Citus Background Task Queue Executor: postgres/marco for (6/79)
2023-01-30 23:22:37.320 CET [13356] LOG: background worker "Citus Background Task Queue Executor: postgres/marco for (6/79)" (PID 15230) exited with exit code 1
2023-01-30 23:22:37.323 CET [15229] LOG: task jobid/taskid failed: 6/79
2023-01-30 23:22:37.323 CET [15229] CONTEXT: Citus Background Task Queue Monitor: postgres
```

According to citus_rebalance_status(), the rebalance remained stuck at the same number of tasks after at least 6 tries.
```
┌────────────────────────────┐
│ jsonb_pretty │
├────────────────────────────┤
│ { ↵│
│ "tasks": [ ↵│
│ ], ↵│
│ "task_state_counts": {↵│
│ "done": 13, ↵│
│ "blocked": 2, ↵│
│ "runnable": 1 ↵│
│ } ↵│
│ } │
└────────────────────────────┘
(1 row)
```
Also it would be nice to reflect errors from pg_dist_background_task in rebalance status.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with citus_rebalance_start and inspect citus_rebalance_status alongside pg_dist_background_task after changing shouldhaveshards. Trace how failed background tasks affect the rebalance state. Done means a rebalance no longer remains stuck after such a failure and its errors are reflected in rebalance status.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, postgresql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.