element-hq / element-hq/synapse
Add background job to clear unreferenced state groups
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
After fixing https://github.com/element-hq/synapse/issues/9406 and https://github.com/element-hq/synapse/issues/17937, we still have a bunch of unreferenced state groups in the DB which point to the full state, causing lots of unnecessary DB usage. We should add a background job to go and delete unreferenced state groups.
Note that we can still have new unreferenced state groups, just that when we purge history we won't de-delta the state group entries and instead delete them.
A one-off background job to delete unreferenced state groups would:
1. Record the current max state group
2. Each run would look at the next N state groups (in ascending order) via `state_groups` table, and check if there are any unreferenced ones. Note that we need to check both `event_to_state_groups` and `state_group_edges` tables.
3. Call [_find_unreferenced_groups](https://github.com/element-hq/synapse/blob/deb09b38362be0c567da166bd697ef8abff422b2/synapse/storage/controllers/purge_events.py#L83) on any unreferenced state groups to get others that can be deleted if the given state group is deleted.
4. Call `mark_state_groups_as_pending_deletion` to schedule them for deletion
I'm also wondering if instead of having this as a one off job we do this periodically to catch new unreferenced state groups.
Contributor guide
Assessment
This issue has not been assessed yet.