Writing results of polygon query as Zarr
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 394
- Forks
- 95
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 7
Description
When attempting to write a subset of a spatialdata object, produced using the polygon_query functionality to disk using the write function, I encountered the following JSON serialization error.
Traceback (most recent call last): [31/1042]
File "/opt/format_conversion.py", line 188, in
main(args.assay, args.data_directory)
File "/opt/format_conversion.py", line 143, in main
sdata.write(XENIUM_ZARR_PATH)
File "/usr/local/lib/python3.10/dist-packages/spatialdata/_core/spatialdata.py", line 1219, in write
self._write_element(
File "/usr/local/lib/python3.10/dist-packages/spatialdata/_core/spatialdata.py", line 1267
, in _write_element
write_points(points=element, group=element_type_group, name=element_name, format=parsed["points"])
File "/usr/local/lib/python3.10/dist-packages/spatialdata/_io/io_points.py", line 73, in w
rite_points
points.to_parquet(path)
File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/core.py", line 5646, in to_pa
rquet
return to_parquet(self, path, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/io/parquet/core.py", line 952
, in to_parquet
i_offset, fmd, metadata_file_exists, extra_write_kwargs = engine.initialize_write(
File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/io/parquet/arrow.py", line 72
4, in initialize_write
inferred_schema = pyarrow_schema_dispatch( [8/1042]
File "/usr/local/lib/python3.10/dist-packages/dask/utils.py", line 772, in call
return meth(arg, *args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/dask/dataframe/backends.py", line 216, in ge
t_pyarrow_schema_pandas
return pa.Schema.from_pandas(obj, preserve_index=preserve_index)
File "pyarrow/types.pxi", line 3148, in pyarrow.lib.Schema.from_pandas
File "/usr/local/lib/python3.10/dist-packages/pyarrow/pandas_compat.py", line 580, in data
frame_to_types
metadata = construct_metadata(
File "/usr/local/lib/python3.10/dist-packages/pyarrow/pandas_compat.py", line 281, in cons
truct_metadata
b'pandas': json.dumps({
File "/usr/lib/python3.10/json/init.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type Scale is not JSON serializable
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce a polygon_query subset followed by sdata.write() and inspect the traceback through spatialdata/_core/spatialdata.py and spatialdata/_io/io_points.py. Check how the points data reaches Dask and PyArrow, then verify that the resulting subset can be written as Zarr without the Scale JSON serialization error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100