apache / apache/datafusion-sqlparser-rs

Add support for DataFusion specific statements

Đang mở
#1,080 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Rust
Star
3.5k
Fork
772
Merge trung bình
4 ngày 9 giờ
Pull request đã merge (30 ngày)
17

Mô tả

**Context**

[arrow-datafusion](https://github.com/apache/arrow-datafusion) currently implements its own parser, [DFParser](https://github.com/apache/arrow-datafusion/blob/d2b3d1c7538b9fb7ab9cfc0c4c6a238b0dcd91e6/datafusion/sql/src/parser.rs#L246-L256) which wraps the parser in this crate in order to parse some DataFusion specific statements.

DataFusion issue: https://github.com/apache/arrow-datafusion/issues/4808

Aiming to upstream this functionality into this crate to remove the parsing code from DataFusion.

**Statements**

Currently two custom statements that DataFusion parses: `COPY TO ...` and `CREATE EXTERNAL TABLE ...`

- There is `EXPLAIN` too, but this is only there to support doing `EXPLAIN` for those new extensions

**COPY TO**

Syntax:

```sql
COPY | ()
TO ''
[ ( key1 value1, key2 value2) ]
```

Examples:

```sql
COPY lineitem TO '/path/to/lineitem.parquet' (format parquet, partitions 16);
COPY (SELECT * FROM lineitem) TO '/path/to/lineitem.parquet';
```

This is extremely similar to the [`COPY` statement from PostgreSQL](https://www.postgresql.org/docs/current/sql-copy.html) that sqlparser-rs already supports, with a few key differences:

1. Doesn't support optional `WITH` keyword
2. Only supports `COPY TO` and not `COPY FROM`
3. Doesn't support column list when source is table
4. Only supports target as string literal, not PROGRAM or STDOUT
5. Options list doesn't constrain keys to a defined set

Points 2, 3 & 4 are non-issues since can inspect the Statement AST to check if want to support this:

https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1463-L1479

Point 1 is a minor issue as the Statement AST above doesn't specify if there was a `WITH` keyword found when parsing, but at the same time DataFusion could just accept this new syntax since this optional keyword has minimal impact.

Point 5 is the major issue, as would either need to modify the fields of the existing `Copy` enum or add a new one specific for DataFusion, since it is a requirement that the keys cannot be constrained (would be parsed as `String`).

**CREATE EXTERNAL TABLE**

Syntax:

```sql
CREATE [ UNBOUNDED ] EXTERNAL TABLE
[ IF NOT EXISTS ]
[ () ]
STORED AS
[ WITH HEADER ROW ]
[ DELIMITER ]
[ COMPRESSION TYPE ]
[ PARTITIONED BY () ]
[ WITH ORDER ()
[ OPTIONS () ]
LOCATION

:= ( , ...)

:= (, ...)

:= ( , ...)

:= ( , ...)
```

Example:

```sql
CREATE UNBOUNDED EXTERNAL TABLE IF NOT EXISTS
kumachan (c1 int)
STORED AS CSV
WITH HEADER ROW
DELIMITER ','
COMPRESSION TYPE zstd
PARTITIONED BY (c1)
WITH ORDER (c1 asc)
OPTIONS (
'k1' 'v1',
'k2' 'v2'
)
LOCATION '/file'
```

This seems to vary heavily from the existing support for `CreateTable`:

https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/ast/mod.rs#L1561-L1604

- Varies in the keyword parsing, such as `WITH HEADER ROW` and `WITH ORDER`

So might need a new statement for this? Or could try to retrofit onto the existing `CreateTable` statement.

**Dialect**

Also will need a new DataFusion dialect to support the above customization (e.g. to be able to toggle between previous/default behaviour for `COPY TO` to parse options as predefined keys, or as generic).

**Alternative**

Instead of adding/modifying as described, could investigate ways to make it easier for downstream consumers to parse their own custom statements.

I see there is this dialect function:

https://github.com/sqlparser-rs/sqlparser-rs/blob/a430d1a5a7bb04bbefd0f2fea07bf25c7fbce8b2/src/dialect/mod.rs#L171-L175

But this doesn't allow for custom statements.

I'm unsure what this could look like, but worth a thought.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Bắt đầu bằng cách so sánh DFParser của DataFusion với các định nghĩa AST Copy và CreateTable hiện có trong src/ast/mod.rs, sau đó kiểm tra điểm mở rộng dialect trong src/dialect/mod.rs. Xác định xem các statement chuyên biệt hay cơ chế custom-statement downstream phù hợp hơn; công việc được xem là hoàn tất khi cú pháp COPY TO và CREATE EXTERNAL TABLE dành riêng cho DataFusion, bao gồm cả hành vi theo dialect, có thể được phân tích mà không cần logic parser trùng lặp trong DataFusion.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
rust, sql
Lĩnh vực
compilers, databases
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Đình trệ
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.