Agent not accepting new connections after system temporarily hit FD limit

#### Overview of the Issue

During a networking problem in our 100+ machine datacenter, the consul agents on 18 of the machines ran into "accept tcp 127.0.0.1:8500: accept4: too many open files in system" errors. 2 of these consul agents then got stuck for hours in a state where they wouldn't accept incoming requests -- attempting to do a KV read using `consul kv get ...` would get this error:

```
Error querying Consul agent: Get http://127.0.0.1:8500/v1/kv/...: dial tcp 127.0.0.1:8500: connect: connection refused
```

The processes themselves were not using large amounts of file descriptors. At the time I noticed after roughly 6 hours, they were only had 16 and 17 FDs open, respectively. But it seems as though the prior `accept` failure got the agents into a bad state somehow.

Around the time that the `accept` requests failed, a variety of "checks" failed as well due to the lack of available FDs, including HTTP checks, TCP checks, and exec/`CheckMonitor` checks. I'm not sure whether one of these failures may have been involved in causing the problem.

Previously when we've seen this problem of refusing new connections (https://github.com/hashicorp/consul/issues/8312#issuecomment-738326472) it was due to an FD leak in the agent itself. This time there was no FD leak involved but the agent still got stuck when it was unable to accept a new connection. I'd hope that new bit of information makes this bug easier to track down now?

Also potentially related to https://github.com/hashicorp/consul/issues/8628, although I'm less familiar with that one.

#### Reproduction Steps

I haven't tried reproducing, and presumably it isn't totally deterministic since only 2 of the 18 agents got stuck when they hit the error, but you could perhaps try by temporarily exhausting the FDs usable by a consul agent and then fixing the exhaustion.

### Consul info for both Client and Server

Both client and server are running v1.6.3.

### Operating system and Environment details

Debian Stretch

### Log Fragments

Goroutine dump from the stuck process:https://gist.github.com/a-robinson/532baca9409ece715732469b127c0547

Start with the linked goroutine dump and the reported accept4 "too many open files" sequence. Reproduce by exhausting file descriptors available to a Consul agent, then clearing the exhaustion, while observing failed checks and incoming requests. Done means the agent resumes accepting connections after the temporary FD limit is removed, with the failure mechanism identified.

Written by the indexing model from the issue text.