parsl.dataflow.errors.JoinError: Join failure for task 1 with failed join dependencies

I am trying to collate csv files from a phenotypic screen into a single parquet or sqlite file using cytotable or cytominer (based on your suggestion). Some details on our study:

I have cellprofiler output in csv format with each folder containing well-level run of cellprofiler across multiple 384-well plates. Each well-level folder contains csv data for segmented nuclei, cells, and cytoplasm using brightfield, NucBlue, and NucGreen with respective file names ending `bf.csv`, `live.csv`, and `(dead|dying).csv`.


I have collated well-level summary data from `Image.csv`. While that works to infer aggregate effects per knockout gene, I like to leverage cell or segment-level data from respective NucBlue (live) and NucGreen (dying or dead) stains, especially for nuclei and less so for cells and cytoplasm (named cyto).

While running cellprofiler, I made a mistake for one of output csv and named it nuclei_dead.csv instead of nuclei_dying.csv. I believe the way collate feature works for cytominer and cytotable is to look for matching string in file name (? and column names for those files), and then start merging data at well and then plate level (depending on folder structure).

For now, I get the following error regardless of if I used original data (with inconsistent filenames) or after renaming _hcssyn_seg_nuclei_dying.csv_ to _hcssyn_seg_nuclei_dead.csv_ (but not replacing dying with dead within column names!). I am not sure if error is related to not replacing dying to dead within column names or something else but would be much help if there is a way to overcome this issue.

I have shared toyset data via email.

>PS: For shared toyset, source path can be one of toyset/orig or toyset/renamed directory

[gist: apptainer definition file to build cytotable_0.0.1p2.sif](https://gist.github.com/sbamin/95647ee7dbe78493565a4fdde104d451), using commit: e74a6785ece152008d91e5f4bd59b8f5fc8b9314

```sh
foo@r209u11n01.mccleary :~/palmer_scratch/syn/toyset$ apptainer run /gpfs/gibbs/pi/lab/foo/hpcenv/opt/apptainer/sifbin/cytotable_0.0.1p2.sif
Python 3.10.11 (main, May 23 2023, 13:58:30) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cytotable import convert
>>> convert(
... source_path="/home/foo/palmer_scratch/syn/toyset/data",
... source_datatype="csv",
... dest_path="take1.parquet",
... dest_datatype="parquet",
... preset="cellprofiler_csv",
... )
```

```
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.10/site-packages/cytotable/convert.py", line 1434, in convert
).result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 301, in handle_exec_update
res = self._unwrap_remote_exception_wrapper(future)
File "/usr/local/lib/python3.10/site-packages/parsl/dataflow/dflow.py", line 573, in _unwrap_remote_exception_wrapper
result.reraise()
File "/usr/local/lib/python3.10/site-packages/parsl/app/errors.py", line 122, in reraise
reraise(t, v, v.__traceback__)
File "/usr/local/lib/python3.10/site-packages/six.py", line 719, in reraise
raise value
File "/usr/local/lib/python3.10/site-packages/parsl/app/errors.py", line 145, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/cytotable/convert.py", line 1094, in _to_parquet
).result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
parsl.dataflow.errors.JoinError: Join failure for task 1 with failed join dependencies from tasks [5]
>>>
```

I am able to successfully run tutorial code with above apptainer image.

```sh
from cytotable import convert

# using a local path with cellprofiler csv presets
convert(
source_path="./tests/data/cellprofiler/ExampleHuman",
source_datatype="csv",
dest_path="ExampleHuman.parquet",
dest_datatype="parquet",
preset="cellprofiler_csv",
)
```