lincc-frameworks / lincc-frameworks/nested-pandas

Support reading FixedShapeTensor pyarrow columns as TensorExtensionArray

Open
#552 0 comments 0 reactions 1 assignee Claimed by @smcguire-cmu View on GitHub
enhancement interface LSDB modalities-support
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

Currently pyarrow FixedShapeTensor columns are mapped to pandas ArrowExtensionArray columns (`pd.ArrowDtype`) by `read_parquet` and `from_pyarrow`, rather than to our TensorExtensionArray.

Suggested change: replace `types_mapper=pd.ArrowDtype` in `from_pyarrow` with a mapper that returns `TensorDtype` for `pa.FixedShapeTensorType` and `pd.ArrowDtype` otherwise. `TensorDtype.__from_arrow__` already handles the rest. Round-trip tests through both the pyarrow and datafusion engines, including a file whose tensor type has an identity permutation (what `pa.FixedShapeTensorArray.from_numpy_ndarray` writes; the dtype normalises it away).

Follows on from #534 and is a prerequisite for #544.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.