airbnb / airbnb/synapse

Synapse fails to restart after zookeeper node disappears

Open
#94 40 comments 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
2.1k
Forks
248
PR merge metrics
No merged PRs in 30d

Description

I am using the Zookeeper watcher to get a list of available nodes - which it successfully does.

However, after a node fails, it does not get removed from haproxy.

Haproxy monitoring page shows those nodes as down, Zookeeper no longer has them as they are ephemeral, but the reconfiguration does not take place.

If I restart Synapse, however, everything goes back to normal.

```
I, [2014-10-09T22:17:20.844370 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovering backends for service proddb
I, [2014-10-09T22:17:20.861672 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovered 6 backends for service proddb
[WARNING] 281/221756 (157) : Server proddb/100.0.10.10:3306_100.0.10.10 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 281/221756 (157) : Server proddb/100.0.10.10:3306_100.0.10.10 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 2ms. 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
I, [2014-10-09T22:18:15.682057 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovering backends for service proddb
I, [2014-10-09T22:18:15.690843 #12] INFO -- Synapse::ZookeeperWatcher: synapse: discovered 5 backends for service proddb
```

Note: Synapse is deployed in Docker and managed by supervisord

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with Synapse::ZookeeperWatcher and the Zookeeper watcher logs showing backend discovery after an ephemeral node disappears; inspect how that discovery triggers HAProxy reconfiguration. Done means a removed node is also removed from HAProxy without restarting Synapse, including in the Docker and supervisord deployment described.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, ruby
Domain
backend, devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.