element-hq / element-hq/synapse

Synapse creates metrics with name collisions

Open
#19,042 0 comments 0 reactions 0 assignees View on GitHub
A-Metrics O-Frequent S-Tolerable T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Problem

There are some metrics which are created in Synapse that use the same metric name. This leads to the metrics being overridden by the latest metric to be registered https://github.com/element-hq/synapse/blob/18232871d0773108a77bb22f31edb5023119886c/synapse/util/metrics.py#L313-L337

So far there are 2 known instances of this happening:
- We instantiate multiple `SyncRestServlet`, one per listener, and in the `__init__` we setup a new `LruCache`.
- We instantiate multiple `ApplicationService` (one per configured application service) which use the `@cached` decorator on some methods.

### Background

This is a long-standing issue and these logs are harmless. Previously, metrics were just being clobbered silently. Now with https://github.com/element-hq/synapse/pull/18828, we have some error logs to call this out.

Contributor guide

Open the contributing guide

Research direction

Start with synapse/util/metrics.py at the linked registration logic, then trace the metric creation paths for SyncRestServlet and ApplicationService. Determine how their per-instance caches produce duplicate names and define a way to keep registrations distinct without the harmless error logs; verify both reported cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.