lincc-frameworks / lincc-frameworks/nested-pandas
Support reading FixedShapeTensor pyarrow columns as TensorExtensionArray
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
Currently pyarrow FixedShapeTensor columns are mapped to pandas ArrowExtensionArray columns (`pd.ArrowDtype`) by `read_parquet` and `from_pyarrow`, rather than to our TensorExtensionArray.
Suggested change: replace `types_mapper=pd.ArrowDtype` in `from_pyarrow` with a mapper that returns `TensorDtype` for `pa.FixedShapeTensorType` and `pd.ArrowDtype` otherwise. `TensorDtype.__from_arrow__` already handles the rest. Round-trip tests through both the pyarrow and datafusion engines, including a file whose tensor type has an identity permutation (what `pa.FixedShapeTensorArray.from_numpy_ndarray` writes; the dtype normalises it away).
Follows on from #534 and is a prerequisite for #544.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.