apache / apache/arrow

[Python] RecordBatchReader.cast() does not work when reading CSV file.

Open
#46,223 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

While investigating pyarrow's capability to read large csv files in streaming fashion I created a simple csv file with a single `mixed_column` column and 1M rows with value `1` and a single last row with value `A`. After I found out type inference happens for the first block only I noticed `RecordBatchReader.cast()` method. While using it does not satisfy my needs because that is being able to infer data types for all record batches separately, I experimented with this `cast()` method and found it seemed not to work.

Reproduction:
```
import pyarrow.csv
import pyarrow.lib
import pyarrow

def cast_test():
with pyarrow.csv.open_csv('/tmp/mixed.csv') as r:
schema = r.schema
updated_schema = schema.set(0, pyarrow.field("mixed_column", pyarrow.string()))
r_cast = r.cast(updated_schema)
for batch in r_cast:
print(batch)
cast_test()
```

Ouptut:
```
$ python scripts/cast_test.py
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
Traceback (most recent call last):
File "/home/pstrzelczak/scripts/cast_test.py", line 12, in
cast_test()
File "/home/pstrzelczak/scripts/cast_test.py", line 10, in cast_test
for batch in r_cast:
File "pyarrow/ipc.pxi", line 671, in pyarrow.lib.RecordBatchReader.__next__
File "pyarrow/ipc.pxi", line 705, in pyarrow.lib.RecordBatchReader.read_next_batch
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: In CSV column #0: CSV conversion error to int64: invalid value 'A'
```

I am using pyarrow 19.0.1 on Ubuntu 24.04 with Python 3.10.

I found this functionality was implemented in https://github.com/apache/arrow/pull/40070 by @paleolimbot

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.