DemocracyLab / DemocracyLab/CivicTechExchange

Recache requests should be outsourced to external job

Open
#709 1 comment 0 reactions 0 assignees View on GitHub
Back End OPERATIONS Performance
Dominant language
JavaScript
Stars
105
Forks
71
PR merge metrics
No merged PRs in 30d

Description

When a heavily-linked entity is updated, triggering a recache among multiple entities, that creates a large cascade of database operations. While we want those linked entities to be updated within minutes of the update operation, we don't need to block the edit operation. This becomes a particular issue when performing large batch operations such as updating projects based on github commits, which can trigger linked entity re-caches for multiple active projects.

## Requirements ##
- Recaching linked entities will now add the requeuing of each entity to a queue that will be processed later
- Queued recaching jobs will have timestamp that will signal to skip the recaching if the entity's last update date is after the requeue timestamp
- Processing job has configurable rate limit

## Resources ##
[Background Tasks in Python](https://devcenter.heroku.com/articles/python-rq)
[DemocracyLab Queue Implementation](https://github.com/DemocracyLab/CivicTechExchange/pull/812)

## Code example ##
For sending email to queue
https://github.com/DemocracyLab/CivicTechExchange/blob/17188e500adb9d7050ad1faf883811353fa4a684/democracylab/emails.py#L470-L471

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.