improve k8s calrissian retry processes

Lando creates a k8s job to run the bioinformatics workflow.
The k8s job runs calrissian within a pod.
By default jobs [retry pod failures 6 times](
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy).
These jobs can run for many hours. If these jobs fail due to their input values retrying them is unlikely succeed. In this scenario retrying a failed calrissian pod just wastes resources. There are scenarios where the problem is outside of the inputs and a retry would be useful, for example a [k8s a node is upgraded and the pod is killed](
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy).

These retries are automatically done and lando isn't involved in them. There are also retries initiated by the user, where lando deletes the current k8s job and creates a new k8s job.

There are also retry issues related to storage. Lando creates a volume for storing temporary output files produced in the workflow. This currently specified via the [--tmp-outdir-prefix calrissian flag](
https://github.com/Duke-GCB/lando/blob/10eb77896e2e4ff0810b6249fb92a303aa966d18/lando/k8s/jobmanager.py#L172)
When the workflow is run again this storage can fill up and lead to [errors due to running out of space](https://github.com/Duke-GCB/calrissian/issues/58#issuecomment-472845101). There is also a `--cache-dir` calrissian flag that may be a better fit.

NOTE: In the version of openshift/kubernetes we are using there are [issues with setting the backoff limit](https://github.com/kubernetes/kubernetes/issues/54870).