apache / apache/datafusion

Support SQL pipe operator

Open
#14,660 6 comments 20 reactions 0 assignees View on GitHub
enhancement PROPOSAL EPIC
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Google BigQuery is releasing support for a pipe operator in their supported Sql dialect. See https://cloud.google.com/bigquery/docs/pipe-syntax-guide and https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/

Databricks seem to be supporting a similar operator https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select-pipeop.html

Relates to https://github.com/apache/datafusion/issues/14514 which is on supporting Sql features that make SQL easier to write and read.

### Describe the solution you'd like

Datafusion supports the pipe operator, which will allow writing queries more ergonomically.

- [x] WHERE https://github.com/apache/datafusion/pull/17278
- [x] ORDER BY https://github.com/apache/datafusion/pull/17278
- [x] LIMIT https://github.com/apache/datafusion/pull/17278
- [x] SELECT https://github.com/apache/datafusion/pull/17278
- [x] EXTEND https://github.com/apache/datafusion/pull/17278
- [ ] SET
- [ ] DROP #20864
- [x] AS https://github.com/apache/datafusion/pull/17312
- [x] AGGREGATE https://github.com/apache/datafusion/pull/17312
- [ ] TABLESAMPLE
- [ ] RENAME
- [x] UNION https://github.com/apache/datafusion/pull/17312
- [x] INTERSECT https://github.com/apache/datafusion/pull/17312
- [x] EXCEPT https://github.com/apache/datafusion/pull/17312
- [ ] CALL
- [ ] PIVOT https://github.com/apache/datafusion/pull/17365
- [ ] UNPIVOT
- [ ] JOIN https://github.com/apache/datafusion/pull/17365

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading the linked PRs 17278, 17312, and 17365 to understand the implemented pipe operators and remaining scope. Review the unchecked SET, TABLESAMPLE, RENAME, CALL, and UNPIVOT items, while noting that DROP has moved to issue #20864. Done means the remaining listed pipe forms are supported in DataFusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.