TimelyDataflow / TimelyDataflow/timely-dataflow
Use cases for timely dataflow
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 293
- Avg merge
- 14h 46m
- Merged PRs (30d)
- 4
Description
I've been working on some data streaming mechanisms from devices (like IoT) into data bases. The on-device part of data generator is implemented in Rust. Currently using MQTT for transport of the data and then held in a queue (rabbitmq or cloud pub/sub).
I'm currently evaluating Apache beam to push data into Google's BigQuery which I've been using already and hugely impressed with. I stumbled across this project as I was looking for a Rust alternative. I haven't fully gone through your documentation or series blog post (apologies!) but before that I have few queries
Am I correct to assume timely is addressing the same use cases as spark or storm or beam like projects?
Also, could you have computations running in separate nodes?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source files or tests. Start with the project documentation and linked series blog posts, then determine whether timely covers the described streaming use cases and multi-node computations; done would be a resolved explanation or documentation update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, rabbitmq, rust, spark
- Domain
- data-engineering, distributed-systems, stream-processing
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100