nomad coping with OS resources limits

When aiming to run many allocations on a single client, it is easy to overlook adjusting the OS resource limits and have nomad hits these limits. As of 0.10.0, Nomad may start user allocations but partially crash before persisting their state, resulting into many leaked processes that Nomad no longer manages. When a client gets to that state, the best option is to destroy client.

Some sample flags for linux:
* file limits configured by ulimit. In production environments, it needs to be set high, e.g. 65K. A sample relevant issue is https://github.com/hashicorp/nomad/issues/3686#issuecomment-372283062 .
* Kernel max pid and thread counts, configured by `/proc/sys/kernel/pid_max` and `/proc/sys/kernel/threads-max` or sysctl. These are global limits to possible running pids/threads in whole system. Nomad will fail to spin up more tasks, but worse may succeed at spinning up the tasks but then crash on spinning up monitoring threads
* https://www.kernel.org/doc/Documentation/sysctl/kernel.txt
* https://stackoverflow.com/questions/39585839/what-are-the-factor-on-which-pid-max-depends-in-linux
* Cgroup/systemd: Specially in systemd environment, user sessions and services may be started inside a PIDS cgroup with restricted maximum tasks that restricts running pids/threads in a similar fashion to the kernel setting, but targeting individual sessions/services rather than globally.
* It's controlled by via systemd unit files `TasksMax`: https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html ; or in cgroups filesystem (e.g. `/sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max`)
* Memory and Swap limits: If processes use more memory than afforded by system, the OOM killer may arbitrary kill important tasks (including nomad agents).
* _TODO_ (any other ones?!)

Hitting these limits is damaging to many other critical services, even if Nomad remained healthy. Docker has some known issues and some guidance for tweaking these values[1][2].

We can try to address this by many means; here is a sample of possible actions:
* Documentation:
* Highlight above limits and any other relevant ones in Guides for productionizing nomad
* Possibly, highlight sample error messages and logs that are related to above
* Issue guidance for monitoring resource exhausion in system and how to best set alerts
* PreFlight Checks:
* Have nomad issue warnings/alerts if limits are low or if a client is close to exhausting
* Resiliency patterns
* Before running a task, reserve resources that are needed for task and fail if not enough
* Handle resources exhaustion and ensure that any started tasks get leaked
* Ensure that a client that consistently fail to start clients is marked unhealthy and no longer is assigned work

[1] https://success.docker.com/article/how-to-reserve-resource-temporarily-unavailable-errors-due-to-tasksmax-setting
[2] https://github.com/docker/for-linux/issues/73

Start by reviewing the productionizing Nomad guidance and the Linux limits named in the issue: ulimit, pid_max, threads-max, TasksMax, cgroup pids.max, and memory or swap limits. Then determine whether the scope is documentation, preflight checks, or client resiliency; the issue lists all three without selecting one. Done would require an agreed scope and corresponding guidance or behavior, but no specific files or tests are identified.

Written by the indexing model from the issue text.