apache / apache/datafusion

Support timestamp formats on CsvReadOptions or Schema

未关闭
#17,548 1 条评论 3 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Rust
星标
9.3k
派生
2.4k
平均合并
3 天 11 小时
30 天内合并 PR
360

描述

### Is your feature request related to a problem or challenge?

In Apache Spark it is possible to define what the expected timestamp format will be.

See `timestampFormat` and `timestampNTZFormat ` [csv options](https://spark.apache.org/docs/latest/sql-data-sources-csv.html)

This works for both user supplied and inferred schemas.

In datafusion similar functionality is only available when parsing CSV via [SQL DDL options](https://datafusion.apache.org/user-guide/sql/format_options.html)

Currently with datafusion that is not possible even for user supplied schemas. Parsing a CSV that has timestamps with non standard formats will result in an error. Example
CSV contains a column `created_at` with the following value `2025-06-23-05.07.34.214000`

Schema definition for the timestamp column:
```
Field::new("created_at", DataType::Timestamp(TimeUnit::Microsecond, None), true),
```
Attempting to parse CSV file
```
Error: ArrowError(ParseError("Error parsing column 11 at line 1: Parser error: Error parsing timestamp from '2025-06-23-05.07.34.214000': invalid timestamp separator"), None)
```

### Describe the solution you'd like

Users should be able to supply a [custom timestamp format](https://docs.rs/chrono/0.4.42/chrono/format/strftime/index.html#specifiers)

### Describe alternatives you've considered

Extend `CsvReadOptions` and allow users to supply a custom timestamp format. This would be similar to the Apache Spark approach. This format would be used to parse timestamps during schema inference or with user supplied schemas.

And, optionally, extend `DataType::Timestamp` to include a user defined timestamp. This would be more flexible as it would allow per column timestamp formats.

### Additional context

- Currenty the only workaround is to define non custom timetamps as being strings and convert them afterwards with extra code.
- It is possible to supply a timestamp format already when parsing with [SQL DDL](https://datafusion.apache.org/user-guide/sql/format_options.html)

贡献指南

打开贡献指南

调研方向

从 CsvReadOptions 和 CSV 时间戳解析路径开始,然后比较现有的 SQL DDL 格式选项。确定自定义格式应如何应用于 schema 推断和用户提供的 schema,包括是否需要更改 DataType::Timestamp;当非标准时间戳值能够通过这两条路径成功解析时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
rust
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。