apache / apache/datafusion

Support "standard" / alternate format arguments for `to_timestamp`

Open
#8,915 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

After https://github.com/apache/arrow-datafusion/pull/8886 (thanks to @Omega359) DataFusion supports converting strings to timestamps using a string format:

```sql
SELEECT to_timestamp('2020-09-08T12:00:00+00:00', '2020-09-08 12/00/00+00:00', '%c', '%+', '%Y-%m-%d %H/%M/%s%#z'
```

Which will parse `'2020-09-08T12:00:00+00:00'` with several possible formats` %c', '%+', '%Y-%m-%d`

However, as @comphead points out, the format used is specific to `chrono` , the underlying Rust library used. These are slightly different semantics than any existing `to_timestamp` (it isn't postgres format strings, nor is it spark format strings, it is something datafusion specific based on the rust chrono format strings)

### Describe the solution you'd like

Ideally users could decide what "dialect" of string format specifiers they wanted to support based on configuration option. For example, either postgres or spark,

However, this is non trivial given the scope of those two implementations

### Describe alternatives you've considered

Users can always use DataFusion's user defined functions to define the semantics they want, for example with a ScalarUDF that rewrites the specified time string from a postgres format into the chrono format

(though there are likely all sorts of corner cases -- see https://github.com/apache/arrow-datafusion/pull/8886#issuecomment-1899745153)

### Additional context

@jhorstmann has notes about Postgres: https://github.com/apache/arrow-datafusion/issues/5398#issuecomment-1820970911
@Omega359 notes that the spark format library is entirely different still: https://github.com/apache/arrow-datafusion/issues/5398#issuecomment-1866926774

Contributor guide

Open the contributing guide

Research direction

Start with the existing to_timestamp entry point and review the format handling introduced by pull request 8886. Read the notes in issues 5398 and 8915 about PostgreSQL and Spark semantics, then determine how a configuration-selected format dialect could be defined and tested; done means the supported dialect behavior and its corner cases are clearly specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.