apache / apache/arrow

[Python] Support pyarrow.Table.cast with CastOptions

Open
#46,138 5 comments 0 reactions 0 assignees View on GitHub
Component: Python Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

According to the documentation it should be possible to pass an `option` argument of type `payrrow.compute.CastOption` to `pyarrow.Table.cast`.

But as far as I can tell:
- I can't figure out how to get it to work
- there's no example in the documentation
- there's no test for it (in `python/pyarrow/tests/test_table.py`)

```python
import pandas as pd
import pyarrow as pa
import pyarrow.compute as pc
import pytest

original_schema = pa.schema([pa.field("timestamp", pa.timestamp("ns", "UTC"))])
table = pa.table(
[[pd.to_datetime("2023-03-15T15:15:00.123456789Z")]], schema=original_schema
)

new_schema = pa.schema([pa.field("timestamp", pa.timestamp("us", "UTC"))])

with pytest.raises(
pa.ArrowInvalid,
match=r"Casting from timestamp\[ns, tz=UTC\] to timestamp\[us, tz=UTC\] would lose data: 1678893300123456789",
):
table.cast(new_schema)

with pytest.raises(
ValueError,
match=r"Must either pass values for 'target_type' and 'safe' or pass a value for 'options'",
):
table.cast(new_schema, options=pc.CastOptions(allow_time_truncate=True))

with pytest.raises(
TypeError,
match=r"Argument 'target_type' has incorrect type \(expected pyarrow.lib.DataType, got pyarrow.lib.Schema\)",
):
table.cast(
new_schema,
options=pc.CastOptions(target_type=new_schema, allow_time_truncate=True),
)

with pytest.raises(
ValueError,
match=r"Must either pass values for 'target_type' and 'safe' or pass a value for 'options'",
):
table.cast(
new_schema,
safe=False,
options=pc.CastOptions(target_type=pa.struct(new_schema), allow_time_truncate=True),
)

```

I'm happy to send an PR, but not sure where the problem is exactly.

PS: as a side note, CastOptions can be converted to bool and be interpreted by mistake as the `safe` argument:
```
assert bool(pc.CastOptions(allow_time_truncate=True)) == True
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.