kubernetes worker pods restart forever

**Describe the bug**
Kubernetes worker pods, perhaps only ones which did not register properly, accumulate forever.
The worker process exits "normally" without an indication in `kubectl logs` that there is some error, but kubernetes immediately restarts that worker (which then fails again).
Nothing causes these workers to go away.

For example, here is one that has been restarted a 1607 times, since it was initially launched over two weeks ago.

```
root@amber:~# minikube kubectl get pods
NAME READY STATUS RESTARTS AGE
funcx-1632329996841 0/1 CrashLoopBackOff 1607 (4m31s ago) 8d
[...]
```

```# minikube kubectl -- logs -f funcx-1632329996841
Collecting funcx-endpoint>=0.2.0
Downloading funcx_endpoint-0.3.3-py3-none-any.whl (91 kB)
```
...
```
Installing collected packages: pycparser, cffi, zipp, urllib3, typing-extensions, six, pyjwt, idna, cryptography, charset-normalizer, certifi, requests, pynacl, importlib-metadata, bcrypt, typeguard, tblib, pyzmq, pyrsistent, pyparsing, psutil, paramiko, lockfile, globus-sdk, docutils, dill, click, attrs, websockets, typer, texttable, python-daemon, py, parsl, packaging, jsonschema, fair-research-login, decorator, configobj, retry, funcx, funcx-endpoint
Successfully installed attrs-21.2.0 bcrypt-3.2.0 certifi-2021.5.30 cffi-1.14.6 charset-normalizer-2.0.6 click-8.0.1 configobj-5.0.6 cryptography-35.0.0 decorator-5.1.0 dill-0.3.4 docutils-0.17.1 fair-research-login-0.2.3 funcx-0.3.3 funcx-endpoint-0.3.3 globus-sdk-2.0.1 idna-3.2 importlib-metadata-4.8.1 jsonschema-4.0.1 lockfile-0.12.2 packaging-21.0 paramiko-2.7.2 parsl-1.1.0 psutil-5.8.0 py-1.10.0 pycparser-2.20 pyjwt-1.7.1 pynacl-1.4.0 pyparsing-2.4.7 pyrsistent-0.18.0 python-daemon-2.3.0 pyzmq-22.3.0 requests-2.26.0 retry-0.9.2 six-1.16.0 tblib-1.7.0 texttable-1.6.4 typeguard-2.12.1 typer-0.4.0 typing-extensions-3.10.0.2 urllib3-1.26.7 websockets-9.1 zipp-3.6.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
PROCESS_WORKER_POOL main event loop exiting normally

```

Crossref parsl https://github.com/Parsl/parsl/issues/2132 -- it's possible/likely that the parsl kubernetes code also has this behaviour.

**To Reproduce**
Start a worker with the python versions incorrectly configured.

**Expected behavior**
Broken worker pods should not accumulate without bound.

**Environment**
my minikube environment on ubuntu

Start by reproducing the failure with a worker started using incorrectly configured Python versions, then inspect the Kubernetes worker behavior and the worker process exit path. Compare the behavior with the linked Parsl Kubernetes issue. Done means broken worker pods no longer accumulate indefinitely and the failure is observable enough to distinguish a bad worker from a healthy one.

Written by the indexing model from the issue text.