cytomining / cytomining/CytoTable
Chunk_size parameter when using with preset + joins variables causes ordering of `Metadata_ImageNumber` column
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
# Problem
When using the `chunk_size` parameter above 1000 with my code, it is causing the "ImageNumber" column in the per_nuclei.parquet file to start from 695 and not 1. This causes the file after concat to only have 695 to the last image number, but it has the correct number of rows for nuclei compared to the SQLite file. I had to change from LEFT JOIN to INNER so that I could get the correct number of rows based on the SQLite file. Using LEFT or RIGHT caused chaos because it was generating extra rows where the nuclei features were all NaN.
Here is the code I was using:
```python
# Only select from the two compartments that we have and merge them
preset_join = """
SELECT
*
FROM
read_parquet('per_image.parquet') as per_image
INNER JOIN read_parquet('per_nuclei.parquet') AS per_nuclei ON
per_nuclei.Metadata_ImageNumber = per_image.Metadata_ImageNumber
"""
for plate_folder in sqlite_dir.iterdir():
output_path = pathlib.Path(f"{output_dir}/{plate_folder.stem}_converted.parquet")
# merge single cells and output as parquet file
convert(
source_path=str(plate_folder),
dest_path=str(output_path),
dest_datatype=dest_datatype,
preset=preset,
joins=preset_join,
parsl_config=local_htex,
chunk_size=10000
)
break
```
# Solution
If I don't use the chunk_size parameter, it looks like it is fixed. But, it goes from running for 1 minute to 8 minutes without the chunk_size increased. I am not sure what is going on at all.
Please feel free to change the title to make it more concise and I am happy to explain and demonstrate with my dataset what is happening.
Contributor guide
Assessment
This issue has not been assessed yet.