Proseg v3 transcript dataframe issue
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
Proseg v3 directly outputs the data as spatialdata zarr-store. The Points dataframe (stored as parquet within the zarr) that contains the transcripts has a column called assignment that stores the cell assignment as integer. However, for transcripts assigned to background this is null. When reading the zarr-store with spatialdata dask/pandas converts this column to float due to the null values.
Theoretically this issue could easily be fixed by changing the dtype_backend in the read_parquet function for the points. However, this will currently fail the validation logic (apparently only numpy dtypes are allowed?) and may have further implications.
This issue does not exist when writing the zarr-store directly via spatialdata as pandas will store a bunch of pandas-specific metadata into the parquet file including the dataype-backend for each column. But given that Proseg writes the dataframe directly from Rust with an Arrow Writer this metadata is not available and integer columns with null will be converted to float when loading it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the points parquet read_parquet path and the validation logic described in the issue, then reproduce loading a Proseg v3 spatialdata zarr-store with a nullable assignment column. Trace the dtype conversion and validation failure; done means background assignments remain nullable integers without breaking supported zarr-store loading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100