apache / apache/datafusion-python

Add support for scalar values with extension types

未关闭
#1,301 3 条评论 0 个 reaction 已指派 1 人 已被 @zaryab-ali 认领 在 GitHub 查看
enhancement
主要语言
Python
星标
604
派生
174
平均合并
1 天 7 小时
30 天内合并 PR
4

描述

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Suppose I have a pyarrow scalar value that contains an extension type. If I try turning that into a literal expression in datafusion, we should get the associated metadata transparently to the user.

Consider this minimal example:

```python
import pyarrow as pa
import uuid
from datafusion import lit

value = pa.scalar(uuid.uuid4().bytes, pa.uuid())

print(lit(value))
```

This currently fails with `ArrowTypeError: Expected bytes, got a 'UUID' object`. That can be overcome with the simple patch

```patch
--- a/src/pyarrow_util.rs
+++ b/src/pyarrow_util.rs
@@ -30,7 +30,11 @@ impl FromPyArrow for PyScalarValue {
fn from_pyarrow_bound(value: &Bound<'_, PyAny>) -> PyResult {
let py = value.py();
let typ = value.getattr("type")?;
- let val = value.call_method0("as_py")?;
+ let val = if value.hasattr("value")? {
+ value.getattr("value")?
+ } else {
+ value.call_method0("as_py")?
+ };
```

But then we still don't have the metadata. It is lost and we get a bare fixed sized binary.

**Describe the solution you'd like**

The above code should *just work*. I have done a little investigation and using the pycapsule interface we *can* get the schema of the array we generate inside `PyScalarValue::from_pyarrow_bound`. We can then plumb this through when calling `lit()`.

Ideally we would take this opportunity to ensure that when we call `PyScalarValue::from_pyarrow_bound` we are also supporting other libraries besides just `pyarrow`. There has been a complaint a few times that we are too tightly coupled to `pyarrow`. In particular it would be good to demonstrate that when converting a Python object that is a scalar value it works for:

- pyarrow
- nanoarrow
- arro3
- polars

I don't think we necessarily need to support pandas since they are not an Arrow library.

**Describe alternatives you've considered**

Alternatively the user can manually turn their data into the underlying storage and then attach the metadata from their extension type. This feels like a poor user experience.

**Additional context**

This came up during a different investigation:

> Also worth evaluating while we're doing this: For scalar values, is it possible for them to contain metadata? If I do `pa.scalar(uuid.uuid4().bytes, type=pa.uuid())` and I check the `type` I should have the extension data. Maybe this is already supported, but as part of this PR I want to evaluate that as well.

_Originally posted by @timsaucer in https://github.com/apache/datafusion-python/issues/1299#issuecomment-3497558869_

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。