cytomining / cytomining/CytoTable
Image features and important Metadata fields missing from parquets
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
Hi there! I have been trying out CytoTable for the first time, following the tutorial for turning CellProfiler analysis CSVs into Parquet files (https://cytomining.github.io/CytoTable/tutorials/cellprofiler_to_parquet.html).
My code:
```
from cytotable import convert
import pandas as pd
import pyarrow.parquet as pq
source_path = "s3://cellpainting-gallery/cpg0037-oasis/broad/workspace/analysis/2025_04_14_OASIS_U2OS_Industry_Batch1/BR00147139/analysis/BR00147139-A01-1/"
convert(
source_path=source_path,
source_datatype="csv",
dest_path="cytotable_kb3",
dest_datatype="parquet",
concat=True,
compartments=("cells", "nuclei", "cytoplasm", "image"),
preset="cellprofiler_csv",
no_sign_request=True,
join=True,
parsl_config=None
)
```
When I compare the columns in this parquet file to the columns in the backends CSV made for this plate by pycytominer collate.py, I see that there are 1300+ columns missing from the parquet file. These include Image measurements (ie Image_Granularity measurements, Image_Texture measurements, etc), `Metadata_Plate`,`Metadata_Well`, `Metadata_Site_Count`,`Metadata_Object_Count`, and all the Counts like `Metadata_Count_Cells`.
The pycytominer-made backends CSV I compared to is here: `s3://cellpainting-gallery/cpg0037-oasis/broad/workspace/backend/2025_04_14_OASIS_U2OS_Industry_Batch1/BR00147139/BR00147139.csv`
I looked through the Cytotable documentation, but I couldn't figure out how to get these metadata and image measurements columns in my Cytotable parquet files. Am I missing a setting or command here? Thanks very much!
Contributor guide
Research direction
Start with the CellProfiler-to-Parquet tutorial and the `cellprofiler_csv` preset used by `convert`, then compare its output with the pycytominer `collate.py` backend CSV. Trace how image measurements, metadata, and count columns are handled for the requested compartments. Done means identifying the relevant setting or limitation and ensuring the expected columns are retained or clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100