Yet another case of state reset in State Resolution v2

This issue has been migrated from [#16629](https://github.com/matrix-org/synapse/issues/16629).

---

### Description

I have encountered yet another [room state reset](https://github.com/matrix-org/synapse/issues/8629).

I suspect that the reset of `m.room.join_rules` was the root cause of the 51 `m.room.member` resets, as the latter would now be rejected events. There is no obvious explanation for the `m.room.join_rules` reset.

No state events were sent or redacted in the time immediately preceding the state reset. The last state event before the reset was sent approximately 2 hours prior.

### Steps to reproduce

1. Use the room as normal.
2. Observe 51 members spontaneously dropped from the room.

### Homeserver

arcticfoxes.net

### Synapse Version

1.96.0rc1

### Installation Method

Debian packages from packages.matrix.org

### Database

PostgreSQL 15.4, single server, not ported, restored a very long time ago

### Workers

Single process

### Platform

- Ubuntu 22.04.3 LTS
- [Hetzner EX101](https://www.hetzner.com/dedicated-rootserver/ex101)
- Intel Core i9-13900
- 64 GB DDR5 ECC
- Dual Samsung PM9A3 1.92 TB NVMe datacenter SSDs in ZFS mirror configuration
- 1 Gbps symmetric internet connection
- Synapse and PostgreSQL run directly on the host OS, no containerization or virtualization involved.

### Configuration

- Presence is enabled.
- The Mjölnir antispam Synapse module is in use.
- [Several experimental features are enabled.](https://github.com/ArcticFoxes-net/Synapse-Ubuntu-ZFS/blob/a4d7a789bd1e17c66f95a3ecc794d38a102f3a65/etc/matrix-synapse/conf.d/experimental_features.yaml)

### Relevant log output

SELECT * FROM current_state_delta_stream WHERE room_id='!frWBWOPyriepLexnsA:arcticfoxes.net' AND stream_id=25794806;

```
stream_id | room_id | type | state_key | event_id | prev_event_id | instance_name
-----------+-------------------------------------+-------------------+-------------------------------------+----------------------------------------------+----------------------------------------------+---------------
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @aldorithms:matrix.org | | $74_XqgpHw_D18w0Xq0Lrc7_nmBvzDx1UajEn9yKc3Qc | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @jhs177992f:arcticfoxes.net | | $rFYpIiFmMMFHYqY_835sCFOEpJVUyLpEDM5Xoqr4uPc | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ariyaro:matrix.org | | $JqoI5tf1YrjoifsPEdxT67IeLJW6BCAdhIsFgxIJQ9Y | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @andreus:matrix.org | | $CUryaK8LXBZxMsDlVohrahm3LaiBC-uCoHHOot0UWS4 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @heliopolis:nitro.chat | | $utSCExwz0EzMAD5eIpCZ9BeEcinoslMFKTjgLO-H6rU | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @backpack:matrix.org | | $cCS3Ry7EFHCXQHTyMrJLratjz0GYRGsyCqU5AkDcnQ4 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @tfox42:matrix.org | | $93-srvPW852lSXze_r-y5VsDsi32Jf6H0GbXxrI8zFo | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @houndvoyager:matrix.org | | $5XMMvwgqdHpX6eyBzamDKpTADycXXvdUhMAIKhtPbTY | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @mercifulrage:converser.eu | | $EDFgLxKC6dz2cG7BYk7gO5xY2NNIKKRcMr7bM9XrPyA | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @crypticangel:pragma-messenger.ch | | $QG5tHgkzolxZAS0E3auilaPTjZqssp5fFr3NPgiwex0 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @coalminecanary:arcticfoxes.net | | $SG_cN3tHXhg9_7_mRHe5Qt4wFlQO7o8hvoV1XPepkTA | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @vovafreeman:matrix.org | | $r9YGaq9q9dbye19fLlO09TO-IC9k6bfOWl-unkJRNNw | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @qoijjj:arcticfoxes.net | | $XbGI2ynAn3WC5hGRrmsBmRbW6f3l5mzzyXoDsqp_C00 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @mebious:tchncs.de | | $zYy_smuirJ0TDeYdC5vgbyQQRxHqxTTF9OTdQ1-cjhs | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @windystone:matrix.org | | $Qa-UYRrnev1cof2voq4VrjlmnDmvEn7cF6QgIYomxFA | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @user0761:matrix.org | | $4w8S40ugloh8ixEufM0tj-_QvnR3oaiXXqdBIPicZDg | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @matrixruinedmylife:arcticfoxes.net | | $Q0mg0Le1GBwXh_H2tFLvwCM97Kf-njC0tW5GcqyFVPU | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @suicidalprincess:nitro.chat | | $Jv4B2FBIDgxkMR93mc9CNE88aTJqG8NrB79tMeVqhJY | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @hat:arcticfoxes.net | | $IDqAum_6ytZX_u0tz-8JglUBjwD39edl329bmExXrzo | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @iustitia:4d2.org | | $86lLB5SWm9xB93vZroSDU4hfjGQ-ZYrJr9fbnE2DZRE | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @beyiyaf450:matrix.org | | $DgRnGVumPcveQGtNWGLKrMzBwowHRJwtrRggmA_ndFo | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @fpmksoba:matrix.org | | $2GtIIACc89vvMrYb1AKJeF0ky4IgDtsZpsFCy_trBTM | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @sneber:nope.chat | | $9mpOqALVoUK1SQ6BbV4H0wKaiooTXbbHnmAIyDjXlvw | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @austin:tchncs.de | | $zg1FQYQk3pTix5UOY7PRxSCmE6Q-94lbXnpgMbByoys | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @bhydden:matrix.org | | $tp1MSrvb4KOEPtxqtJNQ6bKRMdzOVs7puBihkkI6d_4 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @masterrank:matrix.org | | $GJ4SZsaZuJg0NOS0GauO9DcNX3Qa1ESTSP-C84YbSYQ | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @w14:matrix.org | | $i-ZGHSTUJuRLjw6y7IZVSxAbxYZFi42AgYK_xkzEeOA | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @cyri:matrix.org | | $hxptXTV6_m2O4ZoP0w69HOIc1FQ22GC-UnJUz6d0HwQ | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ilove9999:matrix.org | | $k3C1PKERqlnX3ulqHe5qzpzZKfTVWQrknX1z5avD2Dc | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @quietlyjaded:matrix.org | | $NRY6fXAazYFaVa0kLpYWIhy2gfqJW8tVCFyZxVul01E | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @green_sheep:matrix.org | | $cGwRLAiQxivw7d3h9uMca5egIkDOhRVgZnD2Gl_zWVk | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @tisktisk:monero.social | | $lZ01gtBU50UhZ8sacAtpp6Ia5h6Gd54jON7o6Rxu9mc | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @loliloli:arcticfoxes.net | | $iPSlyT37ecv2ZqGH4yOAhkJ6_C51_J-muduqoeSf7PU | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @chili1803:matrix.org | | $jUwmF9DuNmdb56KzAP_D364QM7aocojUY-O95j6hx0c | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @sclanchy:matrix.org | | $27hLXs_qKe4WIgGqy6GigIV6TuknsBmzDSgBHn4c3TI | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @faxex:matrix.org | | $tz3qjuYJnJTHQBaUotnXXVQOEeoUAyGIQTd-1sdhKXc | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ghr89sb:matrix.org | | $UgsO3HsBuDnYVpxzpda-7WUH5vh9AujnN7dEcXX7L5A | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @poisonivy:arcticfoxes.net | | $pVH8QOPHfNfuv1cILOOSELlOuLAPKbliYYbflMkOg-g | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @johan942:matrix.org | | $jjKcZyb7w2DcbYvaKWf5c_052nutBDxcMbM3FJI5V5o | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @carlfriedrich:tchncs.de | | $CeMBICUBs6UPB1hpKooo7YjiTsbevuzC7IRjxihiOxU | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @2nami:matrix.org | | $MEz7DkYNjmX1dXW4aJoiGN8kkNnQ1zjmGekNh3fQijU | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @un32:matrix.org | | $JAAIV6FTLNgANKHyi1dhxskos-RngW_0tr44roqbRL4 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @ancloadi:matrix.org | | $Dw7r7MLjd3JqLzzVdHKuXe0eVsO4wXQzNpOUvoCGtAQ | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @losintos:matrix.org | | $Viw3Yw_2fyjF6Jnk4-ozYMOq6JWrt2TasbEhagy7jyE | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @xyhhx:arcticfoxes.net | | $F6xBiS5WgeyYB7-R4B8eOMKWOD9LA0yNjGj_mDDJEX0 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.join_rules | | $AxE3gEzTLPT_hy7ML1MJKsX7DI6XkqujWP50LTDBODM | $P2FwR8kpzW1LDIZU2lKRc_wmghbXlIQZpwvE5D82d5Q | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @dahustlah:matrix.org | $8GW8hJvAHAGPTQIsPysF_mePLpd-GtZyqYIb4VzfoiU | $aGradtN72do5IOaG1dvRExtW4PAwFfjdlTNZVyOP6sQ | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @gvl7bcq6dn2xrbl:matrix.org | $_s6-w6-v0tE3-tYkWYQ1bbukUs844bn_9bUkttYCXF0 | $iNV97F-d1DnI0dGqj7L23a2pTtRmvSCNPpfDfGbGbg4 | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @masayuki:wired.rehab | $yMVmIFtT_hpeMh-9GLSdY84NrNYSRhn8rUX2Z2AD5_g | $egJIEy0PhxdUfdDCAPAVJDSrIYTLYtdAIIWCQ3zvsNA | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @matchboxbananasynergy:matrix.org | $A4R_VVl2TaN53EWiLBoipPyigkEZodgWOiceeieshaQ | $s-1rvX3ldLsL5Azxj7mSjkhWxboqaEuQiJlMCglMIYk | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @skull_gap:archoslinux.cz | $MX2EhPV0TYzMxegHKJX1U1KKvqTrAp8vwS5DwqqDlNw | $_0SGO3HGR2RgYWYAxczIkwXzCl7e3PLuZgrO_HTgj6Y | master
25794806 | !frWBWOPyriepLexnsA:arcticfoxes.net | m.room.member | @slate-module:matrix.org | $xM6Y-mUyvW9yOZf_lODJOdqLI0Ao1nYuaYQYonysi84 | $2IUW5LONVOx_k1I5nxRv9AuF8BFAp5NL8I_KxFa16Zk | master
(52 rows)
```

### Anything else that would be useful to know?

In addition to arcticfoxes.net documented above, this state reset was confirmed via the C‑S API on matrix.org (Synapse 1.96.0rc1) and a third server (Synapse 1.95.1). I do not know the exact details of their deployments.