apache / apache/arrow

[C++] Unable to read date64 or date32 in specific format from CSV

Open
#28,303 9 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

when importing csv data with dates in the format `"%d-%b-%y"` or `"%d-%b-%Y"` an error is given in conversion:

example:
```python

import pyarrow as pa
from pyarrow import csv

data = b"a,b\n1,15-OCT-15\n2,18-JUN-90\n"
tp = ["%d-%b-%y"]

try:
schema_d64 = pa.schema([pa.field("a", pa.int64()), pa.field("b", pa.date64())])
co_d64 = csv.ConvertOptions(timestamp_parsers=tp, column_types=schema_d64)
a_d64 = csv.read_csv(pa.py_buffer(data), convert_options=co_d64)
except Exception as e:
print(e)
try:
schema_d32 = pa.schema([pa.field("a", pa.int64()), pa.field("b", pa.date32())])
co_d32 = csv.ConvertOptions(timestamp_parsers=tp, column_types=schema_d32)
a_d32 = csv.read_csv(pa.py_buffer(data), convert_options=co_d32)
except Exception as e:
print(e)
```
 

**Reporter**: [Stephen Bias](https://issues.apache.org/jira/browse/ARROW-12539)
#### Related issues:
- [[C++] Parse time32 from string and infer in CSV reader](https://github.com/apache/arrow/issues/27146) (is related to)

**Note**: *This issue was originally created as [ARROW-12539](https://issues.apache.org/jira/browse/ARROW-12539). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproduction using pyarrow.csv.read_csv, ConvertOptions(timestamp_parsers=...), and date32/date64 column types. Trace how the C++ CSV reader converts the supplied date formats, then verify that both "%d-%b-%y" and "%d-%b-%Y" work for the example data without conversion errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.