Where to find possible streaming formats?
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
I am trying to create a structured streaming example that could use .NET Spark to analyze incoming Tweets (i.e. filter them, maybe apply ML.NET sentiment analysis efficiently). I found a nuget package called [Tweetinvi](https://github.com/linvi/tweetinvi) that allows me to create a stream of incoming Tweets.
When I was trying to find a way to connect a Twitter stream with .NET Spark, I wasn't sure the best option. The only input options for DataStreamReader.Format() that I could find in the examples in this repo were "[Kafka](https://github.com/dotnet/spark/blob/e2c1ae13b537bcd8dfd10a50b563a8fedb0b3dbc/examples/Microsoft.Spark.CSharp.Examples/Sql/Streaming/StructuredKafkaWordCount.cs#L38)" and "[socket](https://github.com/dotnet/spark/blob/e2c1ae13b537bcd8dfd10a50b563a8fedb0b3dbc/examples/Microsoft.Spark.CSharp.Examples/Sql/Streaming/StructuredNetworkWordCount.cs#L40)."
Is there a comprehensive list of all possible formats we can use in a .NET Spark structured streaming scenario, or are socket and Kafka the only two options? Is there any explanation of what counts as a socket connection (i.e. if I find host/port information through Tweetinvi, could I just use the socket format for my Spark program)?
If these are the only two format options, or if there is no format suitable for the stream format I have currently generated, I figure I may need to add another step, like finding a way to intertwine my Twitter stream with Kafka, and then create a Kafka-based Spark streaming scenario.
Contributor guide
Assessment
This issue has not been assessed yet.