apache / apache/datafusion-sqlparser-rs
Add support for DataFusion specific statements
- Langage dominant
- Rust
- Étoiles
- 3.5k
- Forks
- 772
- Merge moyen
- 4 j 9 h
- PR mergées (30 j)
- 17
Description
**Context**
[arrow-datafusion](https://github.com/apache/arrow-datafusion) currently implements its own parser, [DFParser](https://github.com/apache/arrow-datafusion/blob/d2b3d1c7538b9fb7ab9cfc0c4c6a238b0dcd91e6/datafusion/sql/src/parser.rs#L246-L256) which wraps the parser in this crate in order to parse some DataFusion specific statements.
DataFusion issue: https://github.com/apache/arrow-datafusion/issues/4808
Aiming to upstream this functionality into this crate to remove the parsing code from DataFusion.
**Statements**
Currently two custom statements that DataFusion parses: `COPY TO ...` and `CREATE EXTERNAL TABLE ...`
- There is `EXPLAIN` too, but this is only there to support doing `EXPLAIN` for those new extensions
**COPY TO**
Syntax:
```sql
COPY | ()
TO ''
[ ( key1 value1, key2 value2) ]
```
Examples:
```sql
COPY lineitem TO '/path/to/lineitem.parquet' (format parquet, partitions 16);
COPY (SELECT * FROM lineitem) TO '/path/to/lineitem.parquet';
```
This is extremely similar to the [`COPY` statement from PostgreSQL](https://www.postgresql.org/docs/current/sql-copy.html) that sqlparser-rs already supports, with a few key differences:
1. Doesn't support optional `WITH` keyword
2. Only supports `COPY TO` and not `COPY FROM`
3. Doesn't support column list when source is table
4. Only supports target as string literal, not PROGRAM or STDOUT
5. Options list doesn't constrain keys to a defined set
Points 2, 3 & 4 are non-issues since can inspect the Statement AST to check if want to support this:
https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1463-L1479
Point 1 is a minor issue as the Statement AST above doesn't specify if there was a `WITH` keyword found when parsing, but at the same time DataFusion could just accept this new syntax since this optional keyword has minimal impact.
Point 5 is the major issue, as would either need to modify the fields of the existing `Copy` enum or add a new one specific for DataFusion, since it is a requirement that the keys cannot be constrained (would be parsed as `String`).
**CREATE EXTERNAL TABLE**
Syntax:
```sql
CREATE [ UNBOUNDED ] EXTERNAL TABLE
[ IF NOT EXISTS ]
[ () ]
STORED AS
[ WITH HEADER ROW ]
[ DELIMITER ]
[ COMPRESSION TYPE ]
[ PARTITIONED BY () ]
[ WITH ORDER ()
[ OPTIONS () ]
LOCATION
:= ( , ...)
:= (, ...)
:= ( , ...)
:= ( , ...)
```
Example:
```sql
CREATE UNBOUNDED EXTERNAL TABLE IF NOT EXISTS
kumachan (c1 int)
STORED AS CSV
WITH HEADER ROW
DELIMITER ','
COMPRESSION TYPE zstd
PARTITIONED BY (c1)
WITH ORDER (c1 asc)
OPTIONS (
'k1' 'v1',
'k2' 'v2'
)
LOCATION '/file'
```
This seems to vary heavily from the existing support for `CreateTable`:
https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1561-L1604
- Varies in the keyword parsing, such as `WITH HEADER ROW` and `WITH ORDER`
So might need a new statement for this? Or could try to retrofit onto the existing `CreateTable` statement.
**Dialect**
Also will need a new DataFusion dialect to support the above customization (e.g. to be able to toggle between previous/default behaviour for `COPY TO` to parse options as predefined keys, or as generic).
**Alternative**
Instead of adding/modifying as described, could investigate ways to make it easier for downstream consumers to parse their own custom statements.
I see there is this dialect function:
https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/dialect/mod.rs#L171-L175
But this doesn't allow for custom statements.
I'm unsure what this could look like, but worth a thought.
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Commencez par comparer le DFParser de DataFusion aux définitions AST existantes de Copy et CreateTable dans src/ast/mod.rs, puis examinez le point d’extension des dialectes dans src/dialect/mod.rs. Déterminez si des statements dédiés ou un mécanisme de custom statements en aval conviennent le mieux ; le travail est considéré comme terminé lorsque la syntaxe spécifique à DataFusion COPY TO et CREATE EXTERNAL TABLE, y compris son comportement selon le dialecte, peut être parsée sans logique de parser dupliquée dans DataFusion.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- rust, sql
- Domaine
- compilers, databases
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100