ooni / ooni/devops

Improve primary keys of clickhouse tables

Open
#410 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
HCL
Stars
4
Forks
15
Avg merge
1h 50m
Merged PRs (30d)
4

Description

Some of the DB tables could have their PRIMARY KEYs changed to improve query performance.

For example the PRIMARY KEY of the obs_web table is this:

ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/ooni/obs_web_repl/{shard}', '{replica}')
PARTITION BY concat(substring(bucket_date, 1, 4), substring(bucket_date, 6, 2))
PRIMARY KEY (measurement_uid, observation_idx)

while the analysis table is this:

ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/ooni/analysis_web_measurement/{shard}', '{replica}')
PARTITION BY substring(measurement_uid, 1, 6)
PRIMARY KEY measurement_uid
ORDER BY (measurement_uid, measurement_start_time, probe_cc, probe_asn, domain)

What this means is that it's triggering very large scans (effectively a full scan), when performing very common filtering operations over the measurement_start_time, probe_cc, probe_asn and domain.

Unfortunately it's not possible to change these with an ALTER query, so instead we are going to have to create the new table and copy the data over.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the ClickHouse definitions for the obs_web and analysis_web_measurement tables shown in the issue. Review their current PRIMARY KEY and ORDER BY choices, then determine replacement keys for the listed filters and plan the new-table data copy. Done means the replacement tables are populated and common measurement_start_time, probe_cc, probe_asn, and domain queries no longer require effectively full scans.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, sql
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.