acmutd / acmutd/portal-backend

Refactor backend to use GCP Pub/Sub

Aberta
#68 0 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
TypeScript
Estrelas
3
Forks
0
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

**GCP Pub/Sub**
We would like to refactor thee backend to support a messaging service. While there are multiple options available we would like to go with GCP Pub/Sub since it will integrate well with many of the other services that are already in use. Read more about [GCP Pub/Sub](https://cloud.google.com/pubsub/docs/overview)

The purpose of this issue to be able to set up message queues that various parts of the backend can pull information from to carry out different tasks. Currently that type of functionality is managed via Firestore document triggers. See [here](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/application/typeform.ts#L100) for an example. This technique essentially relies on code being run whenever we insert a document into Firestore. While functional, it does not quite offer the full range of capabilities that we would like & suffers from significant latency issues.

In an ideal situation we would gain the following benefits from Pub/Sub
- Message queue that allows us to deposit information for completing tasks async
- Being able to review messages posted and see whether they have been pulled / their tasks completed
- Be able to set up efficient pipelines where different services can acknowledge reading/acting on message content (would allow for situations where when we need to perform a series of steps we can keep track of what stage something is in, ideally this is something that we would use step functions for but rip that's an AWS only thing)
- Much lower latency, document based triggers used now are typically used for "eventual consistency" which results in functions not being immediately executed. Pub/Sub would allow for functions to get a message "pushed" to it which results in execution right away.

Our first goal with this is to just migrate existing document trigger based functionality to using Pub/Sub model.

**Task List**
- [ ] Setup up GCP Pub/Sub on acm-core
- [ ] Create message topics for typeform
- [ ] Update the backend logic that [saves typeform responses to firestore](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/application/typeform.ts#L83) to also post a message to pub/sub
- [ ] Update backend logic that has a [firestore document trigger](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/application/typeform.ts#L100) to instead have a pub/sub trigger from the previous step

**Additional Context**
At a later point in time we will probably split out multiple layers of logic that occurs in one of these document triggers into multiple services that all pull from Pub/Sub. For example refer [here](https://github.com/acmutd/portal-backend/blob/e068039f55fa8eaf5a7f0171c3a66cb24b7b7337/functions/src/custom/form.ts#L96) and see that there's 4-5 various steps that execute. These could be pipelined / parallelized into distinct services once Pub/Sub is implemented. Just something to keep in mind while implementing for a scalable design.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Direção de pesquisa

Inspect `functions/src/application/typeform.ts` first: the save flow around line 83 and the current Firestore document trigger around line 100. Then review `functions/src/custom/form.ts` around line 96 to see downstream processing currently chained by triggers. A contributor should first map current trigger-based execution and design equivalent Pub/Sub topics/subscriptions. Completion looks like typeform writes publish to Pub/Sub and the previous trigger logic runs from a Pub/Sub-triggered handler instead of Firestore document triggers.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
gcp, typescript
Domínio
backend, cloud, distributed-systems
Tipo de issue
Refatoração
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Razoavelmente clara
Facilidade para iniciantes
44/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.