apache / apache/datafusion

support more date inferences in the csv reader

Open
#9,331 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

given a csv
```csv
mm/dd/yyyy,mm-dd-yyyy,dd/mm/yyyy,dd-mm-yyyy,yyyy/mm/dd,yyyy-mm-dd
01/01/2012,01-01-2012,01/01/2012,01-01-2012,2012/01/01,2012-01-01
02/02/2013,02-02-2013,02/02/2013,02-02-2013,2013/02/02,2013-02-02
03/03/2014,03-03-2014,03/03/2014,03-03-2014,2014/03/03,2014-03-03
04/04/2015,04-04-2015,04/04/2015,04-04-2015,2015/04/04,2015-04-04
05/05/2016,05-05-2016,05/05/2016,05-05-2016,2016/05/05,2016-05-05
06/06/2017,06-06-2017,06/06/2017,06-06-2017,2017/06/06,2017-06-06
07/07/2018,07-07-2018,07/07/2018,07-07-2018,2018/07/07,2018-07-07
```
currently only the last column will get converted to a `Date32`

```sql
❯ SELECT
arrow_typeof("mm/dd/yyyy") AS "mm/dd/yyyy",
arrow_typeof("mm-dd-yyyy") AS "mm-dd-yyyy",
arrow_typeof("dd/mm/yyyy") AS "dd/mm/yyyy",
arrow_typeof("dd-mm-yyyy") AS "dd-mm-yyyy",
arrow_typeof("yyyy/mm/dd") AS "yyyy/mm/dd",
arrow_typeof("yyyy-mm-dd") AS "yyyy-mm-dd"
from 'testdata/csv/dates.csv' limit 1;
+------------+------------+------------+------------+------------+------------+
| mm/dd/yyyy | mm-dd-yyyy | dd/mm/yyyy | dd-mm-yyyy | yyyy/mm/dd | yyyy-mm-dd |
+------------+------------+------------+------------+------------+------------+
| Utf8 | Utf8 | Utf8 | Utf8 | Utf8 | Date32 |
+------------+------------+------------+------------+------------+------------+
```

### Describe the solution you'd like

it'd be nice to auto detect more of these formats. I thought at the very least `YYYY/MM/DD` or `DD/MM/YYYY` would be valid as they are both extremely common.

### Describe alternatives you've considered

there really arent any good alternatives as there aren't any ergonomic ways to cast the utf8 field to date32 through the sql interface.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the CSV example and the SQL type inspection query shown in the report. Trace the CSV reader's existing date inference for the yyyy-mm-dd column and determine how the requested formats should be distinguished, especially where day and month are ambiguous. Done means the intended formats are inferred consistently and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.