acmutd / acmutd/portal-backend
Refactor backend to use GCP Pub/Sub
- Vorherrschende Sprache
- TypeScript
- Sterne
- 3
- Forks
- 0
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
**GCP Pub/Sub**
We would like to refactor thee backend to support a messaging service. While there are multiple options available we would like to go with GCP Pub/Sub since it will integrate well with many of the other services that are already in use. Read more about [GCP Pub/Sub](https://cloud.google.com/pubsub/docs/overview)
The purpose of this issue to be able to set up message queues that various parts of the backend can pull information from to carry out different tasks. Currently that type of functionality is managed via Firestore document triggers. See [here](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/application/typeform.ts#L100) for an example. This technique essentially relies on code being run whenever we insert a document into Firestore. While functional, it does not quite offer the full range of capabilities that we would like & suffers from significant latency issues.
In an ideal situation we would gain the following benefits from Pub/Sub
- Message queue that allows us to deposit information for completing tasks async
- Being able to review messages posted and see whether they have been pulled / their tasks completed
- Be able to set up efficient pipelines where different services can acknowledge reading/acting on message content (would allow for situations where when we need to perform a series of steps we can keep track of what stage something is in, ideally this is something that we would use step functions for but rip that's an AWS only thing)
- Much lower latency, document based triggers used now are typically used for "eventual consistency" which results in functions not being immediately executed. Pub/Sub would allow for functions to get a message "pushed" to it which results in execution right away.
Our first goal with this is to just migrate existing document trigger based functionality to using Pub/Sub model.
**Task List**
- [ ] Setup up GCP Pub/Sub on acm-core
- [ ] Create message topics for typeform
- [ ] Update the backend logic that [saves typeform responses to firestore](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/application/typeform.ts#L83) to also post a message to pub/sub
- [ ] Update backend logic that has a [firestore document trigger](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/application/typeform.ts#L100) to instead have a pub/sub trigger from the previous step
**Additional Context**
At a later point in time we will probably split out multiple layers of logic that occurs in one of these document triggers into multiple services that all pull from Pub/Sub. For example refer [here](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/custom/form.ts#L96) and see that there's 4-5 various steps that execute. These could be pipelined / parallelized into distinct services once Pub/Sub is implemented. Just something to keep in mind while implementing for a scalable design.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.