Helper task aggregation counters: implement batched background updates
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 70
- Forks
- 16
- Avg merge
- 4h 36m
- Merged PRs (30d)
- 30
Description
Helper task aggregation counters are currently updated in a separate DB transaction from the rest of the aggregation writes, in order to avoid unnecessary write contention. Each update is started in its own background (tokio) task, and will use its own DB transaction.
During a recent incident, we saw this transaction failing due to failure to retrieve a DB connection from the pool. In the event of overload, we could queue up an arbitrary number of these tasks, each awaiting a DB transaction -- there is no limiting factor to the number of background tasks we would spawn.
Instead, we should switch to a model where we have a single background task that receives & occasionally writes updates to the task aggregation counters, similar to how application-level metrics are handled for report uploads. This would cap the number of tasks & DB transactions used to update to 1. And this background task would also be able to batch writes to the counters, as well as handling retries in the case that the counters cannot be written.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing helper task aggregation counter updates and the application-level metrics handling for report uploads, which the issue identifies as the model. Confirm how background tasks and transactions are currently created. Done means one background task receives updates, batches counter writes, limits database transactions to one active task, and retries failed writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100