Processes fail with KeyError

I think this issue (#190) was incorrectly marked closed. I am still having it in version 1.5.3. Seems to happen more in some projects than in others, and doesn't seem to depend how close I am to number_of_final_designs. I am running multiple processes of the same project in parallel, so I wonder if two of them writing to the failure_csv.csv at the same time cause trouble. I can delete failure_csv.csv and continue the jobs, it will make some progress before failing again.

Here's an example of an error traceback; the exact KeyError can vary:

Traceback (most recent call last):
File "/data/jupiter/anaconda3/envs/BindCraft/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 7096, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'ShapeComplementarity'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data/jupiter/programs/BindCraft/bindcraft.py", line 423, in
failure_df[base_column] = failure_df[base_column] + 1
File "/data/jupiter/anaconda3/envs/BindCraft/lib/python3.10/site-packages/pandas/core/frame.py", line 4113, in getitem
indexer = self.columns.get_loc(key)
File "/data/jupiter/anaconda3/envs/BindCraft/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3819, in get_loc
raise KeyError(key) from err
KeyError: 'ShapeComplementarity'

And here's what failure_csv.csv looks like before I delete it and restart the jobs

Unnamed: 0,i_pTM,i_pAE,Trajectory_logits_pLDDT,Trajectory_Clashes,pLDDT,Trajectory_final_pLDDT,Trajectory_one-hot_pLDDT,Trajectory_softmax_pLDDT

Start at bindcraft.py around line 423 and inspect how failure_csv.csv is read and updated. Reproduce the failure with multiple processes writing the file, then verify that concurrent runs no longer produce a KeyError and that failure_csv.csv retains the columns needed for updates.