apache / apache/datafusion-sqlparser-rs

Time travel support

Open
#2,006 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.5k
Forks
772
Avg merge
4d 9h
Merged PRs (30d)
17

Description

Several databases and data lakes support time travel which allows historical versions of a table to be queried. Regrettably, syntax and capabilities do not seem to be terribly consistent across the various implementations.

## Querying Historical Times

This option lets a database be queried as it existed at a certain point in time. (*Question:* Is this "last version prior to the given time"? or is it an error if no snapshot exists with that exact time?)

### [Databricks](https://docs.databricks.com/gcp/en/delta/history#delta-time-travel-syntax)

```
SELECT * FROM people10m TIMESTAMP AS OF '2018-10-18T22:15:12.013Z';
```

### [Snowflake](https://docs.snowflake.com/en/sql-reference/constructs/at-before)

```
SELECT * FROM my_table AT(TIMESTAMP => 'Wed, 26 Jun 2024 09:20:00 -0700'::TIMESTAMP_LTZ);
```

### [Cockroach](https://www.cockroachlabs.com/blog/time-travel-queries-select-witty_subtitle-the_future/)

```
SELECT * FROM t AS OF SYSTEM TIME '2016-06-15 12:45:00'
```

### [DuckLake](https://ducklake.select/docs/stable/duckdb/usage/time_travel.html)

```
SELECT * FROM tbl AT (TIMESTAMP => now() - INTERVAL '1 week');
```

## Querying Specific versions

This options lets a specific "version" of a database to be queried. What exactly this means seems to depend on the database implementation in question.

### Databricks

```
SELECT * FROM people10m VERSION AS OF 123;
```

### Snowflake

Note: unclear if this is the same thing (specific versions) or different

```
SELECT * FROM my_table BEFORE(STATEMENT => '8e5d0ca9-005e-44e6-b858-a8f5b37c5726');
```

### Cockroach

Unsupported

### DuckLake

```
SELECT * FROM tbl AT (VERSION => 3);
```

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository files, tests, or entry points are identified in the issue. Start by comparing the Databricks, Snowflake, Cockroach, and DuckLake syntax and resolve whether historical-time and version queries should share semantics or remain dialect-specific. Done should include an agreed scope for the supported syntax and behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.