Test switching from threads to processes

We had a number of problems with runs hanging during the 1.0 push. A common issue had a stack trace like this:

Thread 0x000079a1b2a006c0 (most recent call first):
File "/usr/local/lib/python3.10/ssl.py", line 1165 in read
File "/usr/local/lib/python3.10/ssl.py", line 1292 in recv
File "/venv/lib/python3.10/site-packages/httpcore/_backends/sync.py", line 128 in read
File "/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 217 in _receive_event
File "/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 177 in _receive_response_headers
File "/venv/lib/python3.10/site-packages/httpcore/_sync/http11.py", line 106 in handle_request
File "/venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 103 in handle_request
File "/venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 236 in handle_request
File "/venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 236 in handle_request
File "/venv/lib/python3.10/site-packages/httpx/_client.py", line 1027 in _send_single_request
File "/venv/lib/python3.10/site-packages/httpx/_client.py", line 991 in _send_handling_redirects
File "/venv/lib/python3.10/site-packages/httpx/_client.py", line 954 in _send_handling_auth
File "/venv/lib/python3.10/site-packages/httpx/_client.py", line 926 in send
File "/venv/lib/python3.10/site-packages/mistralai/basesdk.py", line 218 in do
File "/venv/lib/python3.10/site-packages/mistralai/utils/retries.py", line 72 in do_request
File "/venv/lib/python3.10/site-packages/mistralai/utils/retries.py", line 176 in retry_with_backoff
File "/venv/lib/python3.10/site-packages/mistralai/utils/retries.py", line 104 in retry
File "/venv/lib/python3.10/site-packages/mistralai/basesdk.py", line 255 in do_request
File "/venv/lib/python3.10/site-packages/mistralai/chat.py", line 122 in complete
File "/venv/lib/python3.10/site-packages/modelgauge/suts/mistral_client.py", line 54 in request
File "/venv/lib/python3.10/site-packages/modelgauge/suts/mistral_sut.py", line 64 in evaluate
File "/venv/lib/python3.10/site-packages/modelbench/benchmark_runner.py", line 300 in handle_item
File "/venv/lib/python3.10/site-packages/modelgauge/pipeline.py", line 211 in run
File "/usr/local/lib/python3.10/threading.py", line 953 in run
File "/usr/local/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/usr/local/lib/python3.10/threading.py", line 973 in _bootstrap

This was discussed here and elsewhere around then.

A core problem is that Python's thread management isn't very good. You can't kill a thread. So although there are other possible solutions to this particular problem, the only thing that eliminates a whole class of problems is to switch to subprocesses. But that may introduce other problems, so we should try it it out first.

Start by tracing the execution flow around modelbench/benchmark_runner.py line 300 and modelgauge/pipeline.py line 211, using the reported hanging stack trace as the failure case. Experiment with subprocess-based execution and compare it with the current threaded behavior. Done means the hang scenario no longer leaves work stuck, while the benchmark runner still completes normally.