DnsWatcher < BaseWatcher

We were trying out this service at our organization in an internal datacenter and we temporarily lost internet connection. Because this is resolving the external address of [airbnb](https://github.com/airbnb/synapse/blob/01a2409a5f1e19742ff9380ce683a54b31a4b7ad/lib/synapse/service_watcher/dns.rb#L18) being required as part of 'ping?' it caused synapse to fail entirely. Perhaps throwing a recoverable error rather than a runtime exception would help prevent the entire cluster from collapsing.

``` bash
I, [2016-02-17T17:34:09.333142 #8263] INFO -- Synapse::Haproxy: synapse: restarted haproxy
E, [2016-02-18T16:58:00.932196 #8263] ERROR -- Synapse::Synapse: synapse: encountered unexpected exception # in main thread
W, [2016-02-18T16:58:00.932830 #8263] WARN -- Synapse::Synapse: synapse: exiting; sending stop signal to all watchers
I, [2016-02-18T16:58:00.932962 #8263] INFO -- Synapse::ZookeeperDnsWatcher: synapse: stopping watcher mongo using default stop handler
W, [2016-02-18T16:58:00.933561 #8263] WARN -- Synapse::ZookeeperDnsWatcher::Zookeeper: synapse: zookeeper watcher exiting
I, [2016-02-18T16:58:00.934565 #8263] INFO -- Synapse::ZookeeperDnsWatcher::Zookeeper: synapse: zookeeper watcher cleaning up
I, [2016-02-18T16:58:00.934961 #8263] INFO -- Synapse::ZookeeperDnsWatcher::Zookeeper: synapse: closing zk connection to 172.16.151.82:2181,172.16.151.86:2181,172.16.151.87:2181
I, [2016-02-18T16:58:00.937347 #8263] INFO -- Synapse::ZookeeperDnsWatcher::Zookeeper: synapse: zookeeper watcher cleaned up successfully
/var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/lib/ruby/gems/2.2.0/gems/synapse-0.12.1/lib/synapse.rb:54:in `block (2 levels) in run': synapse: service watcher mongo failed ping! (RuntimeError)
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/lib/ruby/gems/2.2.0/gems/synapse-0.12.1/lib/synapse.rb:53:in `each'
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/lib/ruby/gems/2.2.0/gems/synapse-0.12.1/lib/synapse.rb:53:in `block in run'
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/lib/ruby/gems/2.2.0/gems/synapse-0.12.1/lib/synapse.rb:52:in `loop'
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/lib/ruby/gems/2.2.0/gems/synapse-0.12.1/lib/synapse.rb:52:in `run'
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/lib/ruby/gems/2.2.0/gems/synapse-0.12.1/bin/synapse:60:in `'
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/.bin/synapse:23:in `load'
from /var/lib/mesos/slaves/20160213-002027-1469517996-5050-555-S4/frameworks/20160204-212329-1385631916-5050-163-0000/executors/thermos-1455730435744-bright-devel-authserve-0-6e52c164-8fcb-4e00-bc33-62523f32ad4c/runs/4810df56-7a1d-4bd1-9c2c-84f2c041a181/sandbox/synapse/.bin/synapse:23:in `'
```

Start with lib/synapse/service_watcher/dns.rb at the external Airbnb address lookup referenced in the issue, then trace how ping? reaches the failure handling in lib/synapse.rb. Reproduce the lost-DNS or lost-internet case and determine the expected recoverable behavior. Done means a DNS failure no longer causes the entire Synapse cluster to exit while the watcher reports the failed ping.

Written by the indexing model from the issue text.