Request to shutdown already dead slave will block new requests for ~30 seconds

The reason for this is that CR will start retrying the kill request to the (already dead) slave while the request to the master is still held open. The request will obviously never succeed, so the shutdown request takes as long as it takes the CR master to give up contacting the slave.

Our network retry logic in this situation is probably too generous. It uses the default params to `retry_on_exception_exponential_backoff` which starts with a delay of 0.1s and doubles until it passes the default limit of 15s. (Since the last doubled interval is 12.8s, we exceed the "limit" by quite a bit.)

Local reproduction steps:

``` bash
$ ./main.py master & ./main.py slave &
$ kill -9 `cat ~/.clusterrunner/.clusterrunner_slave.pid`
$ time ./main.py shutdown -m localhost:43000 -s 1
[2016-01-13 17:27:28.150] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 0.1 seconds.
[2016-01-13 17:27:28.260] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 0.2 seconds.
[2016-01-13 17:27:28.468] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 0.4 seconds.
[2016-01-13 17:27:28.881] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 0.8 seconds.
[2016-01-13 17:27:29.689] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 1.6 seconds.
[2016-01-13 17:27:31.298] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 3.2 seconds.
[2016-01-13 17:27:34.510] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 6.4 seconds.
[2016-01-13 17:27:40.927] 41939 WARNING MasterTornadoTh decorators Call to Network.post raised ConnectionError("HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"). Retrying in 12.8 seconds.
[2016-01-13 17:27:53.736] 41939 ERROR MasterTornadoTh cluster_master_ Exception occurred during request to /v1/slave/shutdown.
Traceback (most recent call last):
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
body=body, headers=headers)
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1090, in request
self._send_request(method, url, body, headers)
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1128, in _send_request
self.endheaders(body)
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1086, in endheaders
self._send_output(message_body)
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 924, in _send_output
self.send(msg)
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 859, in send
self.connect()
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 106, in connect
conn = self._new_conn()
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 90, in _new_conn
(self.host, self.port), self.timeout, *extra_args)
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 509, in create_connection
raise err
File "/usr/local/Cellar/python3/3.4.2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 500, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/adapters.py", line 327, in send
timeout=timeout
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 543, in urlopen
raise MaxRetryError(self, url, e)
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1346, in _when_complete
callback()
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/tornado/web.py", line 1367, in _execute_method
self._when_complete(method(*self.path_args, **self.path_kwargs),
File "/Users/jharrington/code/ClusterRunner/app/util/decorators.py", line 63, in function_with_auth
return function(self, *args, **kwargs)
File "/Users/jharrington/code/ClusterRunner/app/web_framework/cluster_master_application.py", line 353, in post
self._cluster_master.set_shutdown_mode_on_slaves(slaves_to_shutdown)
File "/Users/jharrington/code/ClusterRunner/app/master/cluster_master.py", line 177, in set_shutdown_mode_on_slaves
self.handle_slave_state_update(slave, SlaveState.SHUTDOWN)
File "/Users/jharrington/code/ClusterRunner/app/master/cluster_master.py", line 168, in handle_slave_state_update
do_transition(slave)
File "/Users/jharrington/code/ClusterRunner/app/master/cluster_master.py", line 185, in _graceful_shutdown_slave
slave.set_shutdown_mode()
File "/Users/jharrington/code/ClusterRunner/app/master/slave.py", line 179, in set_shutdown_mode
self.kill()
File "/Users/jharrington/code/ClusterRunner/app/master/slave.py", line 192, in kill
self._network.post_with_digest(kill_url, {}, Secret.get())
File "/Users/jharrington/code/ClusterRunner/app/util/network.py", line 60, in post_with_digest
error_on_failure=error_on_failure)
File "/Users/jharrington/code/ClusterRunner/app/util/decorators.py", line 38, in function_with_retries
return function(*args, **kwargs)
File "/Users/jharrington/code/ClusterRunner/app/util/network.py", line 48, in post
return self._request('POST', *args, **kwargs)
File "/Users/jharrington/code/ClusterRunner/app/util/network.py", line 122, in _request
resp = self._session.request(method, url, data=data_to_send, *args, **kwargs)
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "/Users/jharrington/.virtualenvs/clusterrunner/lib/python3.4/site-packages/requests/adapters.py", line 375, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)
[2016-01-13 17:27:53.752] 41939 ERROR MasterTornadoTh tornado.access 500 POST /v1/slave/shutdown (127.0.0.1) 26210.82ms
Unhandled exception handler caught exception.
Traceback (most recent call last):
File "./main.py", line 290, in main
subcommand_class().run(**parsed_args)
File "/Users/jharrington/code/ClusterRunner/app/subcommands/shutdown_subcommand.py", line 18, in run
client.graceful_shutdown_slaves_by_id(slave_ids)
File "/Users/jharrington/code/ClusterRunner/app/client/cluster_api_client.py", line 179, in graceful_shutdown_slaves_by_id
return self._graceful_shutdown_slaves({'slaves': slave_ids})
File "/Users/jharrington/code/ClusterRunner/app/client/cluster_api_client.py", line 197, in _graceful_shutdown_slaves
error_on_failure=True
File "/Users/jharrington/code/ClusterRunner/app/util/network.py", line 60, in post_with_digest
error_on_failure=error_on_failure)
File "/Users/jharrington/code/ClusterRunner/app/util/decorators.py", line 38, in function_with_retries
return function(*args, **kwargs)
File "/Users/jharrington/code/ClusterRunner/app/util/network.py", line 48, in post
return self._request('POST', *args, **kwargs)
File "/Users/jharrington/code/ClusterRunner/app/util/network.py", line 125, in _request
format(url, str(resp.status_code), resp.text))
app.util.network._RequestFailedError: Request to http://localhost:43000/v1/slave/shutdown failed with status_code 500 and response "{"error": "HTTPConnectionPool(host='jharrington.local', port=43001): Max retries exceeded with url: /v1/kill (Caused by : [Errno 61] Connection refused)"}"
./main.py shutdown -m localhost:43000 -s 1 0.23s user 0.07s system 1% cpu 26.620 total
```