[Python] Table.append_column cannot append non-null type column with only null values
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
In pyarrow, I want to append a column of type float (nullable) to a table. If I specify the field, I expect `append_column` to create a new float column.
If all values are `None`, I get a `pyarrow.lib.ArrowInvalid: Field type did not match data type` execption. But `None` should be a valid value for nullable float.
For example, I expect the following
```python
t = pyarrow.table({"A": [1.0, 2.0]})
t2 = t.append_column(
pyarrow.field("B", pyarrow.float64(), nullable=True),
[[None, None]],
)
print(t2)
# pyarrow.Table
# A: double
# B: double
# ----
# A: [[1,2]]
# B: [[null,null]]
```
but I get the error
```
t2 = t.append_column(
pyarrow.field("B", pyarrow.float64(), nullable=True), [[None, None]]
)
File "pyarrow/table.pxi", line 2493, in pyarrow.lib._Tabular.append_column
File "pyarrow/table.pxi", line 5403, in pyarrow.lib.Table.add_column
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Field type did not match data type
```
for comparison, these work but don't do what I want:
```python
t.append_column("B", [[None, None]]) # results in null column [2 nulls]
t.append_column("B", [[1.0, None]]) # results in float column [[1, null]]
```
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.