Support "standard" / alternate format arguments for `to_timestamp`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
After https://github.com/apache/arrow-datafusion/pull/8886 (thanks to @Omega359) DataFusion supports converting strings to timestamps using a string format:
```sql
SELEECT to_timestamp('2020-09-08T12:00:00+00:00', '2020-09-08 12/00/00+00:00', '%c', '%+', '%Y-%m-%d %H/%M/%s%#z'
```
Which will parse `'2020-09-08T12:00:00+00:00'` with several possible formats` %c', '%+', '%Y-%m-%d`
However, as @comphead points out, the format used is specific to `chrono` , the underlying Rust library used. These are slightly different semantics than any existing `to_timestamp` (it isn't postgres format strings, nor is it spark format strings, it is something datafusion specific based on the rust chrono format strings)
### Describe the solution you'd like
Ideally users could decide what "dialect" of string format specifiers they wanted to support based on configuration option. For example, either postgres or spark,
However, this is non trivial given the scope of those two implementations
### Describe alternatives you've considered
Users can always use DataFusion's user defined functions to define the semantics they want, for example with a ScalarUDF that rewrites the specified time string from a postgres format into the chrono format
(though there are likely all sorts of corner cases -- see https://github.com/apache/arrow-datafusion/pull/8886#issuecomment-1899745153)
### Additional context
@jhorstmann has notes about Postgres: https://github.com/apache/arrow-datafusion/issues/5398#issuecomment-1820970911
@Omega359 notes that the spark format library is entirely different still: https://github.com/apache/arrow-datafusion/issues/5398#issuecomment-1866926774
Contributor guide
Research direction
Start with the existing to_timestamp entry point and review the format handling introduced by pull request 8886. Read the notes in issues 5398 and 8915 about PostgreSQL and Spark semantics, then determine how a configuration-selected format dialect could be defined and tested; done means the supported dialect behavior and its corner cases are clearly specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100