apache / apache/datafusion-sqlparser-rs

Add support for DataFusion specific statements

Open
#1,080 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.5k
Forks
772
Avg merge
4d 9h
Merged PRs (30d)
17

Description

**Context**

[arrow-datafusion](https://github.com/apache/arrow-datafusion) currently implements its own parser, [DFParser](https://github.com/apache/arrow-datafusion/blob/d2b3d1c7538b9fb7ab9cfc0c4c6a238b0dcd91e6/datafusion/sql/src/parser.rs#L246-L256) which wraps the parser in this crate in order to parse some DataFusion specific statements.

DataFusion issue: https://github.com/apache/arrow-datafusion/issues/4808

Aiming to upstream this functionality into this crate to remove the parsing code from DataFusion.

**Statements**

Currently two custom statements that DataFusion parses: `COPY TO ...` and `CREATE EXTERNAL TABLE ...`

- There is `EXPLAIN` too, but this is only there to support doing `EXPLAIN` for those new extensions

**COPY TO**

Syntax:

```sql
COPY | ()
TO ''
[ ( key1 value1, key2 value2) ]
```

Examples:

```sql
COPY lineitem TO '/path/to/lineitem.parquet' (format parquet, partitions 16);
COPY (SELECT * FROM lineitem) TO '/path/to/lineitem.parquet';
```

This is extremely similar to the [`COPY` statement from PostgreSQL](https://www.postgresql.org/docs/current/sql-copy.html) that sqlparser-rs already supports, with a few key differences:

1. Doesn't support optional `WITH` keyword
2. Only supports `COPY TO` and not `COPY FROM`
3. Doesn't support column list when source is table
4. Only supports target as string literal, not PROGRAM or STDOUT
5. Options list doesn't constrain keys to a defined set

Points 2, 3 & 4 are non-issues since can inspect the Statement AST to check if want to support this:

https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1463-L1479

Point 1 is a minor issue as the Statement AST above doesn't specify if there was a `WITH` keyword found when parsing, but at the same time DataFusion could just accept this new syntax since this optional keyword has minimal impact.

Point 5 is the major issue, as would either need to modify the fields of the existing `Copy` enum or add a new one specific for DataFusion, since it is a requirement that the keys cannot be constrained (would be parsed as `String`).

**CREATE EXTERNAL TABLE**

Syntax:

```sql
CREATE [ UNBOUNDED ] EXTERNAL TABLE
[ IF NOT EXISTS ]
[ () ]
STORED AS
[ WITH HEADER ROW ]
[ DELIMITER ]
[ COMPRESSION TYPE ]
[ PARTITIONED BY () ]
[ WITH ORDER ()
[ OPTIONS () ]
LOCATION

:= ( , ...)

:= (, ...)

:= ( , ...)

:= ( , ...)
```

Example:

```sql
CREATE UNBOUNDED EXTERNAL TABLE IF NOT EXISTS
kumachan (c1 int)
STORED AS CSV
WITH HEADER ROW
DELIMITER ','
COMPRESSION TYPE zstd
PARTITIONED BY (c1)
WITH ORDER (c1 asc)
OPTIONS (
'k1' 'v1',
'k2' 'v2'
)
LOCATION '/file'
```

This seems to vary heavily from the existing support for `CreateTable`:

https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1561-L1604

- Varies in the keyword parsing, such as `WITH HEADER ROW` and `WITH ORDER`

So might need a new statement for this? Or could try to retrofit onto the existing `CreateTable` statement.

**Dialect**

Also will need a new DataFusion dialect to support the above customization (e.g. to be able to toggle between previous/default behaviour for `COPY TO` to parse options as predefined keys, or as generic).

**Alternative**

Instead of adding/modifying as described, could investigate ways to make it easier for downstream consumers to parse their own custom statements.

I see there is this dialect function:

https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/dialect/mod.rs#L171-L175

But this doesn't allow for custom statements.

I'm unsure what this could look like, but worth a thought.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing DataFusion's DFParser with the existing Copy and CreateTable AST definitions in src/ast/mod.rs, then inspect the dialect extension point in src/dialect/mod.rs. Determine whether dedicated statements or a downstream custom-statement mechanism best fits; done means the DataFusion-specific COPY TO and CREATE EXTERNAL TABLE syntax, including their dialect behavior, can be parsed without DataFusion's duplicate parser logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
compilers, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.