Feature request: Custom nomad agent health checks

In some situations the current heartbeat check is not sufficient to detect problems with nodes running the nomad agent. It would be nice if we could extend the heartbeat check with custom checks.

I had a few cases where an application on a node misbehaved and my node checks in consul went into a failed state. Nomad's heartbeat check didn't detect the problem and just kept scheduling tasks to the node. For now the only way to work around this is by adding a consul watch/handler which starts draining the troublesome node.

@schmichael indicated that there will be some improvements regarding draining and node health detection in v0.8, but no custom health checks.

Start by reviewing Nomad's existing heartbeat and node-draining behavior, then compare it with the Consul node checks and watch/handler workaround described in the issue. The issue names no files, tests, or settled interface; done would require agreeing on how custom checks integrate with node health and scheduling.

Written by the indexing model from the issue text.