citus_rebalance_start gets stuck after changing shouldhaveshards
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
After changing the shouldhaveshards of a node that was receiving shard moves, moves kept failing:
```
2023-01-30 23:22:37.296 CET [13401] LOG: found scheduled background tasks, starting new background task queue monitor
2023-01-30 23:22:37.296 CET [13401] CONTEXT: Citus maintenance daemon for database 5 user 10
2023-01-30 23:22:37.309 CET [15229] LOG: task jobid/taskid started: 6/79
2023-01-30 23:22:37.309 CET [15229] CONTEXT: Citus Background Task Queue Monitor: postgres
2023-01-30 23:22:37.317 CET [15230] ERROR: Moving shards to a node that shouldn't have a shard is not supported
2023-01-30 23:22:37.317 CET [15230] HINT: Allow shards on the target node via SELECT * FROM citus_set_node_property('localhost', 1502, 'shouldhaveshards', true);
2023-01-30 23:22:37.317 CET [15230] CONTEXT: Citus Background Task Queue Executor: postgres/marco for (6/79)
2023-01-30 23:22:37.320 CET [13356] LOG: background worker "Citus Background Task Queue Executor: postgres/marco for (6/79)" (PID 15230) exited with exit code 1
2023-01-30 23:22:37.323 CET [15229] LOG: task jobid/taskid failed: 6/79
2023-01-30 23:22:37.323 CET [15229] CONTEXT: Citus Background Task Queue Monitor: postgres
```
According to citus_rebalance_status(), the rebalance remained stuck at the same number of tasks after at least 6 tries.
```
┌────────────────────────────┐
│ jsonb_pretty │
├────────────────────────────┤
│ { ↵│
│ "tasks": [ ↵│
│ ], ↵│
│ "task_state_counts": {↵│
│ "done": 13, ↵│
│ "blocked": 2, ↵│
│ "runnable": 1 ↵│
│ } ↵│
│ } │
└────────────────────────────┘
(1 row)
```
Also it would be nice to reflect errors from pg_dist_background_task in rebalance status.
Contributor guide
Research direction
Start by reproducing the issue with citus_rebalance_start and inspect citus_rebalance_status alongside pg_dist_background_task after changing shouldhaveshards. Trace how failed background tasks affect the rebalance state. Done means a rebalance no longer remains stuck after such a failure and its errors are reflected in rebalance status.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, postgresql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100