[RFC] Supporting Queues / Messaging Pipelines across Cloud Providers (Ex: AWS SQS)
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 931
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 4
Description
## Feature Request
A commonly used feature in Cloud Platforms is their message-passing/queueing services, such as
* [AWS](https://aws.amazon.com/sqs/): SQS
* [Azure](https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted): [Service-Bus, Storage Queues]
* [GCP](https://cloud.google.com/pubsub/docs/choosing-pubsub-or-cloud-tasks): [Cloud Tasks, Cloud Pub-Sub]
These are often used in conjunction with their other services, such as Storage (Ex: to send a message-notification automatically when a new file is uploaded to a Container/Bucket) and Compute (Ex: To pass information between different compute instances, or to take an action whenever a new file is uploaded to Storage). However, it is a pain right now to deal with the different CSP's syntaxes for using these queues, and would be convenient if we could centralize how we carry out at least basic operations like:
1. Read `n` messages from queue
2. Add new messages into the queue
3. Flush the queue / Delete `n` items from the queue
Contributor guide
Research direction
The issue names no repository files, tests, or entry points. Start by locating Libcloud's provider abstractions and existing cross-provider resource APIs, then compare the basic queue operations listed for AWS, Azure, and GCP. Done should include an agreed design and support for reading, adding, and deleting messages across the targeted providers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, azure, gcp, python
- Domain
- cloud, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100