ooni / ooni/backend

Monitor clickhouse proxy for connection issues

Open
#975 0 comments 0 reactions 1 assignee View on GitHub

@LDiazN is already working on this.

Since Jul 24, 2025.

Dominant language
Python
Stars
61
Forks
33
Avg merge
3d 2h
Merged PRs (30d)
3

Description

Right now the Clickhouse proxy is a tier 0 component, all Clickhouse traffic in AWS must go through the Clickhouse proxy. If this component fails, every service in AWS using Clickhouse will fail.

One possible failure could be that the Clickhouse proxy is redeployed and the firewall rules in data1 are not updated to match the new ip address of the redeployed server

We need a good monitoring to detect issues with the clickhouse proxy as swift as possible, particularly with the connection between Clickhouse and the proxy

A possible solution could be:

  1. Add a timer to the clickhouse proxy to ping the database once every couple minutes
  2. Log the result of the ping to grafana
  3. Add alerts to grafana for:
    4. Failed pings: The server did not respond or responded with an error
    5. Missing pings: The server is not sending ping status, meaning that maybe the entire machine is down

Note that this becomes increasingly important as we move more components from the monolith to AWS. See: https://github.com/ooni/backend/issues/807#issuecomment-3103559446

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.