apache / apache/datafusion

Document streaming usecase (like `UNBOUNDED` tables)

Open
#9,016 18 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Someone asked in [discord](https://discord.com/channels/885562378132000778/1166447479609376850/1200557140096594053):

> I'm looking at Datafusion and Polars as potential solutions for calculating averages over a sliding window of events, where the window is bound by event time. I've just come across Datafusion, would anyone be able to clarify if it's suitable for this use case? In essence, I have events streaming in via RPC that I want to feed into a a system that gives the above outcome.

I am pretty sure this is exactly the case for using `UNBOUNDED` tables with explicitly defined `ORDER BY` from [Synnada](https://www.synnada.ai/) and [Arroyo](https://github.com/ArroyoSystems/arroyo) others. However, when I went to look for the documentation, I could't find any mention of this usecase or documentation of unbounded tables

### Describe the solution you'd like

I would like to help make it easier for people to use DataFusion for streaming usecases by:
- [ ] Documenting the `UNBOUNDED` keyword [in the `CREATE EXTERNAL TABLE` documentation](https://arrow.apache.org/datafusion/user-guide/sql/ddl.html#create-external-table)
- [ ] Add an example in https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples/examples with a simple streaming example (perhaps implementing some simple version of the use case described in the description)
- [ ] Add a [section to the library guide](https://arrow.apache.org/datafusion/library-user-guide/index.html) giving some basic overview

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the CREATE EXTERNAL TABLE documentation and the existing examples under datafusion-examples/examples, then review the library guide structure. Document the UNBOUNDED keyword and add a simple streaming example based on the described event-time sliding-window use case. Done means the DDL documentation, example, and library-guide overview clearly explain this use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.