element-hq / element-hq/synapse

Streams tables are never cleared out

Open
#5,888 2 comments 0 reactions 0 assignees View on GitHub
A-Database A-Disk-Space T-Enhancement z-p2
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#5888](https://github.com/matrix-org/synapse/issues/5888).

---

A number of 'stream' tables are appended-to, but are never cleared out. Such tables include:
* ~~cache_invalidation_stream_by_instance~~ (fixed by https://github.com/matrix-org/synapse/pull/15868)
* current_state_delta_stream
* ~~presence_stream~~ (does get cleaned out; has been ever since its creation in https://github.com/matrix-org/synapse/pull/582)
* ~~device_inbox~~ (not really a 'stream' table actually, so a separate problem: #3599)
* device_federation_outbox
* ~~local_group_updates~~ (removed in https://github.com/matrix-org/synapse/pull/12967)
* push_rules_stream
* ex_outlier_stream
* ~public_room_list_stream~ (now removed)
* `device_lists_changes_in_room` (probably not really a stream table, #13043)

Contributor guide

Open the contributing guide

Research direction

Start by auditing the remaining tables listed in the issue: current_state_delta_stream, device_federation_outbox, push_rules_stream, and ex_outlier_stream. Trace how each table is written and whether any cleanup path exists; done requires an agreed, safe retention or cleanup approach for the tables that still accumulate rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.