element-hq / element-hq/synapse

synapse_http_server_response_time_seconds_bucket has too high cardinality

Open
#11,082 0 comments 0 reactions 0 assignees View on GitHub
A-Metrics T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#11082](https://github.com/matrix-org/synapse/issues/11082).

---

### Description

This issue was discovered while looking at other problematic time series due to #11081. There's a similar cardinality issue here, caused by the multiplying factor between the `code`, `servlet` and `tag` labels on the series.

Also, it's problematic to have time series that are part of the same bucket but don't all have the same set of labels. There's only a few series that have the `tag` label. That should probably be a separate histogram as the label difference will cause aggregation issues.

### Steps to reproduce

- Run lots of Synapse instances
- Scrape them all with a single Prometheus instance
- Prometheus gets sad

### Version information

- **Homeserver**: any
- **Version**: any since this metric got introduced
- **Install method**: unrelated
- **Platform**: unrelated

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.