Synapse fails to restart after zookeeper node disappears
- Dominant language
- Ruby
- Stars
- 2.1k
- Forks
- 248
- PR merge metrics
- No merged PRs in 30d
Description
I am using the Zookeeper watcher to get a list of available nodes - which it successfully does.
However, after a node fails, it does not get removed from haproxy.
Haproxy monitoring page shows those nodes as down, Zookeeper no longer has them as they are ephemeral, but the reconfiguration does not take place.
If I restart Synapse, however, everything goes back to normal.
```
I, [2014-10-09T22:17:20.844370 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovering backends for service proddb
I, [2014-10-09T22:17:20.861672 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovered 6 backends for service proddb
[WARNING] 281/221756 (157) : Server proddb/100.0.10.10:3306_100.0.10.10 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 281/221756 (157) : Server proddb/100.0.10.10:3306_100.0.10.10 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
I, [2014-10-09T22:18:15.682057 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovering backends for service proddb
I, [2014-10-09T22:18:15.690843 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovered 5 backends for service proddb
```
Note: Synapse is deployed in Docker and managed by supervisord
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with Synapse::ZookeeperWatcher and the Zookeeper watcher logs showing backend discovery after an ephemeral node disappears; inspect how that discovery triggers HAProxy reconfiguration. Done means a removed node is also removed from HAProxy without restarting Synapse, including in the Docker and supervisord deployment described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, ruby
- Domain
- backend, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100