element-hq / element-hq/synapse

Add background job to clear unreferenced state groups

Open
#18,150 13 comments 7 reactions 0 assignees View on GitHub
T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

After fixing https://github.com/element-hq/synapse/issues/9406 and https://github.com/element-hq/synapse/issues/17937, we still have a bunch of unreferenced state groups in the DB which point to the full state, causing lots of unnecessary DB usage. We should add a background job to go and delete unreferenced state groups.

Note that we can still have new unreferenced state groups, just that when we purge history we won't de-delta the state group entries and instead delete them.

A one-off background job to delete unreferenced state groups would:
1. Record the current max state group
2. Each run would look at the next N state groups (in ascending order) via `state_groups` table, and check if there are any unreferenced ones. Note that we need to check both `event_to_state_groups` and `state_group_edges` tables.
3. Call [_find_unreferenced_groups](https://github.com/element-hq/synapse/blob/deb09b38362be0c567da166bd697ef8abff422b2/synapse/storage/controllers/purge_events.py#L83) on any unreferenced state groups to get others that can be deleted if the given state group is deleted.
4. Call `mark_state_groups_as_pending_deletion` to schedule them for deletion

I'm also wondering if instead of having this as a one off job we do this periodically to catch new unreferenced state groups.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.