Kubernetes Coexecution jobs are only removed under some circumstances

Such as:

- When the user stops/deletes the job output(s)
- When the job ends in error (because the Pulsar runner's `fail_job()` calls `stop_job()`

Circumstances where the job is not removed:

- When the job finishes normally
- When the job hits its (k8s) walltime and is killed by k8s

This last one is a source of job "loss" (stuck non-terminal) because Pulsar will never send a terminal status update. The runner should probably poll (as in galaxyproject/galaxy#9911) for this case.

The quickest and easiest (and IMO correct) solution would be to set the TTL in the template [as described in the docs](https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs). But it would also be a good idea to call `MessageCoexecutionPodJobClient.kill()` for all jobs when their terminal message is received.