apache / apache/arrow

[Python] Table.append_column cannot append non-null type column with only null values

Open
#46,918 3 comments 0 reactions 1 assignee Claimed by @GujaLomsadze View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In pyarrow, I want to append a column of type float (nullable) to a table. If I specify the field, I expect `append_column` to create a new float column.
If all values are `None`, I get a `pyarrow.lib.ArrowInvalid: Field type did not match data type` execption. But `None` should be a valid value for nullable float.

For example, I expect the following
```python
t = pyarrow.table({"A": [1.0, 2.0]})

t2 = t.append_column(
pyarrow.field("B", pyarrow.float64(), nullable=True),
[[None, None]],
)

print(t2)
# pyarrow.Table
# A: double
# B: double
# ----
# A: [[1,2]]
# B: [[null,null]]
```

but I get the error

```
t2 = t.append_column(
pyarrow.field("B", pyarrow.float64(), nullable=True), [[None, None]]
)
File "pyarrow/table.pxi", line 2493, in pyarrow.lib._Tabular.append_column
File "pyarrow/table.pxi", line 5403, in pyarrow.lib.Table.add_column
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Field type did not match data type
```

for comparison, these work but don't do what I want:
```python
t.append_column("B", [[None, None]]) # results in null column [2 nulls]
t.append_column("B", [[1.0, None]]) # results in float column [[1, null]]
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.