Document streaming usecase (like `UNBOUNDED` tables)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Someone asked in [discord](https://discord.com/channels/885562378132000778/1166447479609376850/1200557140096594053):
> I'm looking at Datafusion and Polars as potential solutions for calculating averages over a sliding window of events, where the window is bound by event time. I've just come across Datafusion, would anyone be able to clarify if it's suitable for this use case? In essence, I have events streaming in via RPC that I want to feed into a a system that gives the above outcome.
I am pretty sure this is exactly the case for using `UNBOUNDED` tables with explicitly defined `ORDER BY` from [Synnada](https://www.synnada.ai/) and [Arroyo](https://github.com/ArroyoSystems/arroyo) others. However, when I went to look for the documentation, I could't find any mention of this usecase or documentation of unbounded tables
### Describe the solution you'd like
I would like to help make it easier for people to use DataFusion for streaming usecases by:
- [ ] Documenting the `UNBOUNDED` keyword [in the `CREATE EXTERNAL TABLE` documentation](https://arrow.apache.org/datafusion/user-guide/sql/ddl.html#create-external-table)
- [ ] Add an example in https://github.com/apache/arrow-datafusion/tree/main/datafusion-examples/examples with a simple streaming example (perhaps implementing some simple version of the use case described in the description)
- [ ] Add a [section to the library guide](https://arrow.apache.org/datafusion/library-user-guide/index.html) giving some basic overview
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the CREATE EXTERNAL TABLE documentation and the existing examples under datafusion-examples/examples, then review the library guide structure. Document the UNBOUNDED keyword and add a simple streaming example based on the described event-time sliding-window use case. Done means the DDL documentation, example, and library-guide overview clearly explain this use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100