typelevel / typelevel/frameless
Type Spark’s Structured Streaming
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Description
We are currently missing these two Dataset method:
- DataStreamWriter writeStream()
- Dataset withWatermark(String eventTime, String delayThreshold)
That require some understanding of Spark streaming to be properly typed and tested. Here is the relevant documentation if anyone is interested and getting started on that:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Structured Streaming programming guide and the Databricks watermarking article linked in the issue, then inspect the existing Dataset and DataStreamWriter typings. Add typed support for writeStream() and withWatermark(String eventTime, String delayThreshold), with tests covering both methods and their Spark Streaming types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100