apache / apache/datafusion-sqlparser-rs
Add support for DataFusion specific statements
- Ngôn ngữ chính
- Rust
- Star
- 3.5k
- Fork
- 772
- Merge trung bình
- 4 ngày 9 giờ
- Pull request đã merge (30 ngày)
- 17
Mô tả
**Context**
[arrow-datafusion](https://github.com/apache/arrow-datafusion) currently implements its own parser, [DFParser](https://github.com/apache/arrow-datafusion/blob/d2b3d1c7538b9fb7ab9cfc0c4c6a238b0dcd91e6/datafusion/sql/src/parser.rs#L246-L256) which wraps the parser in this crate in order to parse some DataFusion specific statements.
DataFusion issue: https://github.com/apache/arrow-datafusion/issues/4808
Aiming to upstream this functionality into this crate to remove the parsing code from DataFusion.
**Statements**
Currently two custom statements that DataFusion parses: `COPY TO ...` and `CREATE EXTERNAL TABLE ...`
- There is `EXPLAIN` too, but this is only there to support doing `EXPLAIN` for those new extensions
**COPY TO**
Syntax:
```sql
COPY | ()
TO ''
[ ( key1 value1, key2 value2) ]
```
Examples:
```sql
COPY lineitem TO '/path/to/lineitem.parquet' (format parquet, partitions 16);
COPY (SELECT * FROM lineitem) TO '/path/to/lineitem.parquet';
```
This is extremely similar to the [`COPY` statement from PostgreSQL](https://www.postgresql.org/docs/current/sql-copy.html) that sqlparser-rs already supports, with a few key differences:
1. Doesn't support optional `WITH` keyword
2. Only supports `COPY TO` and not `COPY FROM`
3. Doesn't support column list when source is table
4. Only supports target as string literal, not PROGRAM or STDOUT
5. Options list doesn't constrain keys to a defined set
Points 2, 3 & 4 are non-issues since can inspect the Statement AST to check if want to support this:
https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1463-L1479
Point 1 is a minor issue as the Statement AST above doesn't specify if there was a `WITH` keyword found when parsing, but at the same time DataFusion could just accept this new syntax since this optional keyword has minimal impact.
Point 5 is the major issue, as would either need to modify the fields of the existing `Copy` enum or add a new one specific for DataFusion, since it is a requirement that the keys cannot be constrained (would be parsed as `String`).
**CREATE EXTERNAL TABLE**
Syntax:
```sql
CREATE [ UNBOUNDED ] EXTERNAL TABLE
[ IF NOT EXISTS ]
[ () ]
STORED AS
[ WITH HEADER ROW ]
[ DELIMITER ]
[ COMPRESSION TYPE ]
[ PARTITIONED BY () ]
[ WITH ORDER ()
[ OPTIONS () ]
LOCATION
:= ( , ...)
:= (, ...)
:= ( , ...)
:= ( , ...)
```
Example:
```sql
CREATE UNBOUNDED EXTERNAL TABLE IF NOT EXISTS
kumachan (c1 int)
STORED AS CSV
WITH HEADER ROW
DELIMITER ','
COMPRESSION TYPE zstd
PARTITIONED BY (c1)
WITH ORDER (c1 asc)
OPTIONS (
'k1' 'v1',
'k2' 'v2'
)
LOCATION '/file'
```
This seems to vary heavily from the existing support for `CreateTable`:
https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1561-L1604
- Varies in the keyword parsing, such as `WITH HEADER ROW` and `WITH ORDER`
So might need a new statement for this? Or could try to retrofit onto the existing `CreateTable` statement.
**Dialect**
Also will need a new DataFusion dialect to support the above customization (e.g. to be able to toggle between previous/default behaviour for `COPY TO` to parse options as predefined keys, or as generic).
**Alternative**
Instead of adding/modifying as described, could investigate ways to make it easier for downstream consumers to parse their own custom statements.
I see there is this dialect function:
https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/dialect/mod.rs#L171-L175
But this doesn't allow for custom statements.
I'm unsure what this could look like, but worth a thought.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Bắt đầu bằng cách so sánh DFParser của DataFusion với các định nghĩa AST Copy và CreateTable hiện có trong src/ast/mod.rs, sau đó kiểm tra điểm mở rộng dialect trong src/dialect/mod.rs. Xác định xem các statement chuyên biệt hay cơ chế custom-statement downstream phù hợp hơn; công việc được xem là hoàn tất khi cú pháp COPY TO và CREATE EXTERNAL TABLE dành riêng cho DataFusion, bao gồm cả hành vi theo dialect, có thể được phân tích mà không cần logic parser trùng lặp trong DataFusion.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- rust, sql
- Lĩnh vực
- compilers, databases
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 35/100