Monitor clickhouse proxy for connection issues
@LDiazN is already working on this.
Since Jul 24, 2025.
- Dominant language
- Python
- Stars
- 61
- Forks
- 33
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 3
Description
Right now the Clickhouse proxy is a tier 0 component, all Clickhouse traffic in AWS must go through the Clickhouse proxy. If this component fails, every service in AWS using Clickhouse will fail.
One possible failure could be that the Clickhouse proxy is redeployed and the firewall rules in data1 are not updated to match the new ip address of the redeployed server
We need a good monitoring to detect issues with the clickhouse proxy as swift as possible, particularly with the connection between Clickhouse and the proxy
A possible solution could be:
- Add a timer to the clickhouse proxy to ping the database once every couple minutes
- Log the result of the ping to grafana
- Add alerts to grafana for:
4. Failed pings: The server did not respond or responded with an error
5. Missing pings: The server is not sending ping status, meaning that maybe the entire machine is down
Note that this becomes increasingly important as we move more components from the monolith to AWS. See: https://github.com/ooni/backend/issues/807#issuecomment-3103559446
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.