[Python] RecordBatchReader.cast() does not work when reading CSV file.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
While investigating pyarrow's capability to read large csv files in streaming fashion I created a simple csv file with a single `mixed_column` column and 1M rows with value `1` and a single last row with value `A`. After I found out type inference happens for the first block only I noticed `RecordBatchReader.cast()` method. While using it does not satisfy my needs because that is being able to infer data types for all record batches separately, I experimented with this `cast()` method and found it seemed not to work.
Reproduction:
```
import pyarrow.csv
import pyarrow.lib
import pyarrow
def cast_test():
with pyarrow.csv.open_csv('/tmp/mixed.csv') as r:
schema = r.schema
updated_schema = schema.set(0, pyarrow.field("mixed_column", pyarrow.string()))
r_cast = r.cast(updated_schema)
for batch in r_cast:
print(batch)
cast_test()
```
Ouptut:
```
$ python scripts/cast_test.py
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
pyarrow.RecordBatch
mixed_column: string
----
mixed_column: ["1","1","1","1","1","1","1","1","1","1",...,"1","1","1","1","1","1","1","1","1","1"]
Traceback (most recent call last):
File "/home/pstrzelczak/scripts/cast_test.py", line 12, in
cast_test()
File "/home/pstrzelczak/scripts/cast_test.py", line 10, in cast_test
for batch in r_cast:
File "pyarrow/ipc.pxi", line 671, in pyarrow.lib.RecordBatchReader.__next__
File "pyarrow/ipc.pxi", line 705, in pyarrow.lib.RecordBatchReader.read_next_batch
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: In CSV column #0: CSV conversion error to int64: invalid value 'A'
```
I am using pyarrow 19.0.1 on Ubuntu 24.04 with Python 3.10.
I found this functionality was implemented in https://github.com/apache/arrow/pull/40070 by @paleolimbot
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.