element-hq / element-hq/synapse
`Gauge` callbacks aren't thread safe
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Problem
RuntimeError: dictionary changed size during iteration
```
Traceback (most recent call last):
File "/home/synapse/.local/share/uv/python/cpython-3.14.6-linux-x86_64_v2-gnu/lib/python3.14/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/exposition.py", line 158, in prometheus_app
status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/exposition.py", line 122, in _bake_output
output = encoder(registry)
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/openmetrics/exposition.py", line 61, in generate_latest
for metric in registry.collect():
~~~~~~~~~~~~~~~~^^
File "/home/synapse/src/synapse/metrics/__init__.py", line 145, in collect
for metric in REGISTRY.collect():
~~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/registry.py", line 94, in collect
yield from collector.collect()
~~~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 91, in collect
for suffix, labels, value, timestamp, exemplar, native_histogram_value in self._samples():
~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 275, in _multi_samples
for suffix, sample_labels, value, timestamp, exemplar, native_histogram_value in metric._samples():
~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 268, in _samples
return self._child_samples()
~~~~~~~~~~~~~~~~~~~^^
File "/home/synapse/env-python314-6/lib/python3.14/site-packages/prometheus_client/metrics.py", line 497, in samples
return (Sample('', {}, float(f()), None, None),)
~^^
File "/home/synapse/src/synapse/util/batching_queue.py", line 121, in
).set_function(lambda: sum(len(q) for q in self._next_values.values()))
~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/synapse/src/synapse/util/batching_queue.py", line 121, in
).set_function(lambda: sum(len(q) for q in self._next_values.values()))
~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: dictionary changed size during iteration
```
Relevant code:
https://github.com/element-hq/synapse/blob/1979fcca52e4fec8eb89bf804145b86d86ab9b89/synapse/util/batching_queue.py#L119-L121
Spotted because these issues popped up in Sentry but they just show some pieces of the error stack,
- https://sentry.tools.element.io/organizations/element/issues/11253798/
- https://sentry.tools.element.io/organizations/element/issues/11253797/
- https://sentry.tools.element.io/organizations/element/issues/11253795/
Then dove into the [logs from `matrix.org`](https://grafana.vpn.infra.matrix.org/explore?schemaVersion=1&panes=%7B%22oow%22%3A%7B%22datasource%22%3A%22P8E80F9AEF21F6940%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22%7Bjob%3D%5C%22synapse%5C%22%7D+%7C%3D+%60sentinel%60%22%2C%22queryType%22%3A%22range%22%2C%22datasource%22%3A%7B%22type%22%3A%22loki%22%2C%22uid%22%3A%22P8E80F9AEF21F6940%22%7D%2C%22editorMode%22%3A%22builder%22%2C%22direction%22%3A%22backward%22%2C%22maxLines%22%3A5000%7D%5D%2C%22range%22%3A%7B%22from%22%3A%221784053025063%22%2C%22to%22%3A%221784053025314%22%7D%2C%22panelsState%22%3A%7B%22logs%22%3A%7B%22logs%22%3A%7B%22visualisationType%22%3A%22logs%22%7D%2C%22sortOrder%22%3A%22Descending%22%7D%7D%2C%22compact%22%3Afalse%7D%7D&orgId=1)
### Potential suspects
Same as https://github.com/element-hq/synapse/issues/18764
I'm guessing the actual problem occurs when using the dedicated `metrics` listener type which runs on a different thread. So when we `collect` from the other thread, sometimes we collide with the dictionary changing size on the main thread.
### Potential solutions
We probably need to use `threading.Lock()` like we do for [`InFlightGauge`](https://github.com/element-hq/synapse/blob/e16fbdcdcc95d626f6d3dcd0b6a85fdb0a06456a/synapse/metrics/__init__.py#L209C7-L350)
Contributor guide
Research direction
Start in synapse/util/batching_queue.py at the Gauge callback around lines 119-121, then trace how registry.collect() runs through the metrics listener. Compare the synchronization used by InFlightGauge in synapse/metrics/__init__.py. Done means concurrent queue updates and metric collection no longer produce the reported dictionary-changed-size RuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100