citusdata / citusdata/pg_shard

pg_shard may fail to mark shard placement as invalid under some circumstances

Open
#101 0 comments 0 reactions 0 assignees View on GitHub
bug research
Dominant language
C
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

The bug happens when pg_shard fails to INSERT to shard placement and postgres is shut down or psql connection is closed before shard placement status is updated.

This is not easy to reproduce bug. But, if a _sleep()_ function call is added to this [line](https://github.com/citusdata/pg_shard/blob/develop/pg_shard.c#L1786), reproducing becomes easy.

Assuming that _sleep()_ is added, the bug can be reproduced with following steps:
1. Create a cluster with 1 master, 2 workers
2. Distribute table and create worker shards with replication factor 2
3. Stop one of the worker nodes
4. Connect to psql, and get its pid, _select pg_backend_pid();_
5. Issue an `INSERT` on that psql session. During the `INSERT` (since we added a sleep, it takes at least _the sleep seconds_), execute shell command "_kill -9 pid_of_psql_"
6. Restart both master and the stopped worker node.
7. Connect to worker nodes and observe that one of the shards is divergent
8. But shard placements on metadata has all `STATE_FINALIZED` status

The main problem here is that we do not execute remote commands and state status changes in an atomic way.

A possible Solution that we can try is to check whether _HOLD_INTERRUPTS()_/_RESUME_INTERRUPTS()_ works. Also, check if these function call pair has any drawbacks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.