cytomining / cytomining/CytoTable

Image features and important Metadata fields missing from parquets

Open
#425 5 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
21
Forks
6
Avg merge
2d 3h
Merged PRs (30d)
6

Description

Hi there! I have been trying out CytoTable for the first time, following the tutorial for turning CellProfiler analysis CSVs into Parquet files (https://cytomining.github.io/CytoTable/tutorials/cellprofiler_to_parquet.html).

My code:
```
from cytotable import convert
import pandas as pd
import pyarrow.parquet as pq

source_path = "s3://cellpainting-gallery/cpg0037-oasis/broad/workspace/analysis/2025_04_14_OASIS_U2OS_Industry_Batch1/BR00147139/analysis/BR00147139-A01-1/"
convert(
source_path=source_path,
source_datatype="csv",
dest_path="cytotable_kb3",
dest_datatype="parquet",
concat=True,
compartments=("cells", "nuclei", "cytoplasm", "image"),
preset="cellprofiler_csv",
no_sign_request=True,
join=True,
parsl_config=None
)
```

When I compare the columns in this parquet file to the columns in the backends CSV made for this plate by pycytominer collate.py, I see that there are 1300+ columns missing from the parquet file. These include Image measurements (ie Image_Granularity measurements, Image_Texture measurements, etc), `Metadata_Plate`,`Metadata_Well`, `Metadata_Site_Count`,`Metadata_Object_Count`, and all the Counts like `Metadata_Count_Cells`.

The pycytominer-made backends CSV I compared to is here: `s3://cellpainting-gallery/cpg0037-oasis/broad/workspace/backend/2025_04_14_OASIS_U2OS_Industry_Batch1/BR00147139/BR00147139.csv`

I looked through the Cytotable documentation, but I couldn't figure out how to get these metadata and image measurements columns in my Cytotable parquet files. Am I missing a setting or command here? Thanks very much!

Contributor guide

Open the contributing guide

Research direction

Start with the CellProfiler-to-Parquet tutorial and the `cellprofiler_csv` preset used by `convert`, then compare its output with the pycytominer `collate.py` backend CSV. Trace how image measurements, metadata, and count columns are handled for the requested compartments. Done means identifying the relevant setting or limitation and ensuring the expected columns are retained or clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.