Speed up ingestion of submissions
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 26
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
Looks like our current setup is not keeping up well with submission bursts now, and latency suffers:
Try to find out how to improve performance. E.g. increase maximum number of messages/objects loaded from the queue in one go, or switch from Cloud Functions to an always-running streaming setup where we continuously receive messages and stream them into BigQuery and PostgreSQL.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the submission queue and the Cloud Functions ingestion path described in the issue, including how messages are loaded and written to BigQuery and PostgreSQL. Compare burst latency for larger batches with an always-running streaming approach. Done means submission bursts are handled with measurably lower latency and the chosen architecture is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, cloud, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100