airbnb / airbnb/synapse

docker watcher - dependent on publicly exposed ports to the parent host.

Open
#109 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
2.1k
Forks
248
PR merge metrics
No merged PRs in 30d

Description

The current implementation is quite restricting. For example using synapse within a docker container, it uses the host IP within @discovery["servers"].

This means you need to publicly expose all containers in the parent host, rather than just using the private ips of the containers themselves. So rather than having discovery, you end up with a fixed mapping based on ports.

When not using publicly exposed ports you end up with this invalid configuration, with a broken port.

```
server 172.17.42.1:_basket-service 172.17.42.1: cookie 172.17.42.1:_basket-service check inter 2s rise 3 fall 2
```

```

"Ports": {
"8080/tcp": null,
"8081/tcp": null
}

```

Error with a little debugging...

```
[ root@52cc33d73359:~ ]$ /usr/bin/ruby1.9.1 /usr/local/bin/synapse -c /synapse.conf.json
I, [2015-01-25T18:22:20.776251 #1971] INFO -- Synapse::Synapse: synapse: starting...
I, [2015-01-25T18:22:20.781601 #1971] INFO -- Synapse::Synapse: synapse: regenerating haproxy config
I, [2015-01-25T18:22:20.781990 #1971] INFO -- Synapse::DnsWatcher: synapse: discovered 1 backends for service http-webservices-member
W, [2015-01-25T18:22:20.782539 #1971] WARN -- Synapse::Haproxy: synapse: no backends found for watcher http-webservices-basket
I, [2015-01-25T18:22:20.783647 #1971] INFO -- Synapse::Haproxy: synapse: could not open haproxy config file at /etc/haproxy/haproxy.cfg
Docker::Container { :id => b8589347f04c9e3a811bf78f49149d645cba9b48afccdd52f64308e039ca3106, :connection => Docker::Connection { :url => http://172.17.42.1:4243, :options => {} } }
8080
{"8080"=>"8080", "8081"=>""}
{"8080"=>"", "8081"=>""}
[{"name"=>"basket-service", "host"=>"172.17.42.1", "port"=>"8080"},
{"name"=>"basket-service", "host"=>"172.17.42.1", "port"=>""}]
I, [2015-01-25T18:22:20.819391 #1971] INFO -- Synapse::DockerWatcher: synapse: discovered 2 backends for service http-webservices-basket
I, [2015-01-25T18:22:20.821997 #1971] INFO -- Synapse::Haproxy: synapse: restarted haproxy
I, [2015-01-25T18:22:20.823559 #1971] INFO -- Synapse::Synapse: synapse: regenerating haproxy config
[ALERT] 024/182222 (2005) : parsing [/etc/haproxy/haproxy.cfg:107] : server 172.17.42.1:_basket-service has neither service port nor check port nor tcp_check rule 'connect' with port information. Check has been disabled.
[ALERT] 024/182222 (2005) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
[ALERT] 024/182222 (2005) : Fatal errors found in configuration.
E, [2015-01-25T18:22:22.844902 #1971] ERROR -- Synapse::Synapse: synapse: encountered unexpected exception # in main thread
W, [2015-01-25T18:22:22.845050 #1971] WARN -- Synapse::Synapse: synapse: exiting; sending stop signal to all watchers
I, [2015-01-25T18:22:22.845184 #1971] INFO -- Synapse::DockerWatcher: synapse: stopping watcher http-webservices-basket using default stop handler
I, [2015-01-25T18:22:22.845321 #1971] INFO -- Synapse::DnsWatcher: synapse: stopping watcher http-webservices-member using default stop handler
/var/lib/gems/1.9.1/gems/synapse-0.11.1/lib/synapse/haproxy.rb:780:in `restart': failed to reload haproxy via service haproxy reload: * Reloading haproxy haproxy (RuntimeError)
...fail!
from /var/lib/gems/1.9.1/gems/synapse-0.11.1/lib/synapse/haproxy.rb:550:in `update_config'
from /var/lib/gems/1.9.1/gems/synapse-0.11.1/lib/synapse.rb:51:in `block in run'
from /var/lib/gems/1.9.1/gems/synapse-0.11.1/lib/synapse.rb:43:in `loop'
from /var/lib/gems/1.9.1/gems/synapse-0.11.1/lib/synapse.rb:43:in `run'
from /var/lib/gems/1.9.1/gems/synapse-0.11.1/bin/synapse:60:in `'
from /usr/local/bin/synapse:23:in `load'
from /usr/local/bin/synapse:23:in `'
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the DockerWatcher behavior described in the issue and the HAProxy failure at lib/synapse/haproxy.rb:780. Reproduce the case with Docker container ports set to null, then verify that generated HAProxy configuration contains valid backend ports and reloads successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, ruby
Domain
backend, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.