Epic: Worker Monitoring

An epic issue to have oversight over monitoring on the worker.

The high level brief is: we need better visibility of what's going on inside the worker, especially when things go wrong.

We should consider metrics tracking, sentry reporting, email notification, grafana, etc.

Related:

#603
#402

Things we want

• When a worker claims something (we do this already in Lightning, but will be useful later to track drift between what lightning thinks and what the worker knows).
• When a worker has to kill a run or job.
• Memory sampling for:
  • At least the whole process tree
  • engine
  • workers? (will need to think about how useful this is on it's own since they are 'disposable processes' and picking them out of a crowd in monitoring may not be that useful).
• CPU usage? (might just be solved by monitoring the pod directly)

We need to figure out the best approach for how to integrate this into prometheus, do we expose an aggregate http service (or use lightning for that) that collects up the metrics?

We probably don't want to use service discovery for monitoring? Do we?
There is an advantage of workers exposing their own /metrics server, makes the worker better for everyone.