Running a Beaker Executor job leaves loads of uncommitted datasets in the workspace

### 🐛 Describe the bug

Run the catwalk training job specified here: https://github.com/allenai/catwalk/commit/5ba019204b0ff36c1c4da7feab4515342e9d9ad2

Command line is `tango --settings experiments/train_all_the_things/tango.yml run experiments/train_all_the_things/train_all_the_things.jsonnet`.

It will run for quite a while. Two jobs will usually fail. I don't know why it's always two.
1. When the run is complete, it tells you that two jobs failed, but you have no way of finding the logs for the failed jobs. (#387)
2. When you run it again, it finds all the succeeded jobs, as it should. The failed ones will start up again, but hang indefinitely somewhere in the Beaker workspace.
3. When you check your workspace, you will see hundreds of uncommitted datasets. Many of these contain actual, successful results. But two of them are the cause for the hang. It's impossible to know which ones to delete so I can finish my run. You can see many of these uncommitted datasets in the ai2/task-complexity workspace.

The actual problem here is the undiagnosable failure of the original jobs, and then the undiagnosable hang (which can be fixed if you remove the right uncommitted datasets). From a user perspective, having all those uncommitted datasets doesn't matter as such, but I suspect that it points to a deeper issue.

### Versions

asd