element-hq / element-hq/synapse

`Gauge` callbacks aren't thread safe

Open
#19,961 0 comments 0 reactions 0 assignees View on GitHub
A-Metrics T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Problem

RuntimeError: dictionary changed size during iteration

```
Traceback (most recent call last):
File "/home/synapse/.local/share/uv/python/cpython-3.14.6-linux-x86_64_v2-gnu/lib/python3.14/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/exposition.py", line 158, in prometheus_app
status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/exposition.py", line 122, in _bake_output
output = encoder(registry)
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/openmetrics/exposition.py", line 61, in generate_latest
for metric in registry.collect():
~~~~~~~~~~~~~~~~^^
File "/home/synapse/src/synapse/metrics/__init__.py", line 145, in collect
for metric in REGISTRY.collect():
~~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/registry.py", line 94, in collect
yield from collector.collect()
~~~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 91, in collect
for suffix, labels, value, timestamp, exemplar, native_histogram_value in self._samples():
~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 275, in _multi_samples
for suffix, sample_labels, value, timestamp, exemplar, native_histogram_value in metric._samples():
~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 268, in _samples
return self._child_samples()
~~~~~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 497, in samples
return (Sample('', {}, float(f()), None, None),)
~^^
File "/home/synapse/src/synapse/util/batching_queue.py", line 121, in
).set_function(lambda: sum(len(q) for q in self._next_values.values()))
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/src/synapse/util/batching_queue.py", line 121, in
).set_function(lambda: sum(len(q) for q in self._next_values.values()))
~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: dictionary changed size during iteration
```

Relevant code:

https://github.com/element-hq/synapse/blob/1979fcca52e4fec8eb89bf804145b86d86ab9b89/synapse/util/batching_queue.py#L119-L121

Spotted because these issues popped up in Sentry but they just show some pieces of the error stack,

- https://sentry.tools.element.io/organizations/element/issues/11253798/
- https://sentry.tools.element.io/organizations/element/issues/11253797/
- https://sentry.tools.element.io/organizations/element/issues/11253795/

Then dove into the [logs from `matrix.org`](https://grafana.vpn.infra.matrix.org/explore?schemaVersion=1&panes=%7B%22oow%22%3A%7B%22datasource%22%3A%22P8E80F9AEF21F6940%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7Bjob%3D%5C%22synapse%5C%22%7D+%7C%3D+%60sentinel%60%22%2C%22queryType%22%3A%22range%22%2C%22datasource%22%3A%7B%22type%22%3A%22loki%22%2C%22uid%22%3A%22P8E80F9AEF21F6940%22%7D%2C%22editorMode%22%3A%22builder%22%2C%22direction%22%3A%22backward%22%2C%22maxLines%22%3A5000%7D%5D%2C%22range%22%3A%7B%22from%22%3A%221784053025063%22%2C%22to%22%3A%221784053025314%22%7D%2C%22panelsState%22%3A%7B%22logs%22%3A%7B%22logs%22%3A%7B%22visualisationType%22%3A%22logs%22%7D%2C%22sortOrder%22%3A%22Descending%22%7D%7D%2C%22compact%22%3Afalse%7D%7D&orgId=1)

### Potential suspects

Same as https://github.com/element-hq/synapse/issues/18764

I'm guessing the actual problem occurs when using the dedicated `metrics` listener type which runs on a different thread. So when we `collect` from the other thread, sometimes we collide with the dictionary changing size on the main thread.

### Potential solutions

We probably need to use `threading.Lock()` like we do for [`InFlightGauge`](https://github.com/element-hq/synapse/blob/e16fbdcdcc95d626f6d3dcd0b6a85fdb0a06456a/synapse/metrics/__init__.py#L209C7-L350)

Contributor guide

Open the contributing guide

Research direction

Start in synapse/util/batching_queue.py at the Gauge callback around lines 119-121, then trace how registry.collect() runs through the metrics listener. Compare the synchronization used by InFlightGauge in synapse/metrics/__init__.py. Done means concurrent queue updates and metric collection no longer produce the reported dictionary-changed-size RuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.