Too many open files

### Nomad version

```
$ nomad -v
Nomad v0.7.0
```

### Operating system and Environment details

```
$ uname -a
Linux iot-useast1-prod-nomad-server-1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
```

```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
```

We have **5** Nomad server nodes and **113** Nomad client nodes on AWS EC2.

### Issue

One of our Nomad server nodes ran out of file descriptors, and now the cluster is struggling to select a leader. This happened the 3rd time already. Previously it was happening on version 0.5.6, and now it's still happening on 0.7.0 after we upgraded.

We can see from the `lsof.log` below that the vast majority (about 75%) of open file descriptors are towards our `nomad-client-admin-4` node, which doesn't run more allocations than other `nomad-client-admin-*` nodes. I included the log for `nomad-client-admin-4` as well, where the only thing I can see is that there is a `nomad_exporter` job which is being restarted frequently, I don't know if that might be the cause.

### Reproduction steps

N/A

### Nomad Server logs (if appropriate)

There are a lot of `too many file descriptors` log lines now, so I tried to extract something relevant:

Earliest errors we have

```
2017/12/16 06:47:59 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:47:59 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:47:59 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:47:59 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:47:59 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:47:59 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:09 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:09 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:09 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:10 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:10 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:10 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:19 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:19 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:19 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:20 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:20 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:20 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:29 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:29 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:29 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:30 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:30 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:30 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:39 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:39 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:39 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:40 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:40 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:40 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:50 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:50 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:48:50 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:51 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:48:51 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:48:51 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:02 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:02 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:02 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:12 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:12 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:12 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:13 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:13 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:13 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:24 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:24 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:24 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:26 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:26 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:26 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:39 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:49:39 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:39 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:41 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:49:41 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:49:41 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:50:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:50:01 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:50:01 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:50:02 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:20 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:20 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:50:20 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:50:22 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:50:22 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:50:22 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:40 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:40 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:50:40 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:50:42 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:50:42 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:50:42 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:50:55.527388 [ERR] worker: failed to dequeue evaluation: eval broker disabled
2017/12/16 06:51:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:51:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:51:00 [ERR] raft: Failed to AppendEntries to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:51:03 [ERR] raft: Failed to heartbeat to 10.0.31.85:4647: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:51:03 [ERR] raft: Failed to heartbeat to 10.0.31.133:4647: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:51:03 [ERR] raft: Failed to heartbeat to 10.0.31.89:4647: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:53:31.974209 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:53:31.974447 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:53:34.493762 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:53:34.518940 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:53:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.0.31.133:4647 10.0.31.133:4647}: dial tcp 10.0.31.133:4647: i/o timeout
2017/12/16 06:53:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.0.31.85:4647 10.0.31.85:4647}: dial tcp 10.0.31.85:4647: i/o timeout
2017/12/16 06:53:44 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.0.31.89:4647 10.0.31.89:4647}: dial tcp 10.0.31.89:4647: i/o timeout
2017/12/16 06:55:43 [ERR] raft-net: Failed to flush response: write tcp 10.0.11.172:4647->10.0.21.37:33314: write: broken pipe
2017/12/16 06:55:44.008619 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:55:44.802281 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:55:45.124885 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:55:45.124895 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:56:48.406489 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:56:48.406509 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
2017/12/16 06:56:52.407520 [ERR] worker: failed to dequeue evaluation: rpc error: rpc error: eval broker disabled
```

Today's errors: [nomad.log](https://github.com/hashicorp/nomad/files/1579057/nomad.log)

`sudo lsof` output: [lsof.log](https://github.com/hashicorp/nomad/files/1579058/lsof.log)

### Nomad Client logs (if appropriate)

`nomad-client-admin-4` log: [nomad-admin-client-4.log](https://github.com/hashicorp/nomad/files/1579101/nomad-admin-client-4.log)

### Job file (if appropriate)

prometheus_exporters.hcl

```
job "prometheus_exporters" {
type = "service"
datacenters = ["useast1"]

constraint {
attribute = "${node.class}"
value = "admin"
}

group "consul_exporter" {
count = 1

task "consul_exporter" {
driver = "docker"
config {
image = "registry.service.m2x:5000/attm2x/consul-exporter:673081d"
args = ["-consul.server=${attr.unique.network.ip-address}:8500"]
port_map { http = 9107 }
}
service {
name = "prometheus-consul-exporter"
port = "http"
check {
type = "http"
path = "/"
timeout = "5s"
interval = "30s"
}
}
resources {
memory = 64
network {
mbits = 1
port "http" {}
}
}
}
}

group "es_iot_exporter" {
count = 1

task "es_iot_exporter" {
driver = "docker"
config {
image = "registry.service.m2x:5000/attm2x/elasticsearch-exporter:2dd77e1"
args = [
"-es.uri=http://244.es-iot-client.service.m2x:9200",
"-es.all"
]
port_map { http = 9108 }
}
service {
name = "prometheus-es-iot-exporter"
port = "http"
check {
type = "http"
path = "/"
timeout = "5s"
interval = "30s"
}
}
resources {
memory = 64
network {
mbits = 1
port "http" {}
}
}
}
}

group "nomad_exporter" {
count = 1

task "nomad_exporter" {
driver = "docker"
config {
image = "registry.service.m2x:5000/attm2x/nomad-exporter:e65b05d"
command = "nomad-exporter"
args = [
"-nomad.server=http://nomad.service.m2x:4646"
]
port_map { http = 9172 }
}
service {
name = "prometheus-nomad-exporter"
port = "http"
check {
type = "http"
path = "/"
timeout = "5s"
interval = "30s"
}
}
resources {
memory = 64
network {
mbits = 1
port "http" {}
}
}
}
}
}
```

Start with the Nomad server logs and lsof.log, comparing the file-descriptor connections to nomad-client-admin-4 with other clients. Review the nomad_exporter job's frequent restarts as a possible lead. Done means identifying a reproducible cause and a verified resolution for the recurring file-descriptor exhaustion and resulting leader-selection failures.

Written by the indexing model from the issue text.