Improve primary keys of clickhouse tables
Nobody has claimed this yet.
- Dominant language
- HCL
- Stars
- 4
- Forks
- 15
- Avg merge
- 1h 50m
- Merged PRs (30d)
- 4
Description
Some of the DB tables could have their PRIMARY KEYs changed to improve query performance.
For example the PRIMARY KEY of the obs_web table is this:
ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/ooni/obs_web_repl/{shard}', '{replica}')
PARTITION BY concat(substring(bucket_date, 1, 4), substring(bucket_date, 6, 2))
PRIMARY KEY (measurement_uid, observation_idx)
while the analysis table is this:
ENGINE = ReplicatedReplacingMergeTree('/clickhouse/{cluster}/tables/ooni/analysis_web_measurement/{shard}', '{replica}')
PARTITION BY substring(measurement_uid, 1, 6)
PRIMARY KEY measurement_uid
ORDER BY (measurement_uid, measurement_start_time, probe_cc, probe_asn, domain)
What this means is that it's triggering very large scans (effectively a full scan), when performing very common filtering operations over the measurement_start_time, probe_cc, probe_asn and domain.
Unfortunately it's not possible to change these with an ALTER query, so instead we are going to have to create the new table and copy the data over.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the ClickHouse definitions for the obs_web and analysis_web_measurement tables shown in the issue. Review their current PRIMARY KEY and ORDER BY choices, then determine replacement keys for the listed filters and plan the new-table data copy. Done means the replacement tables are populated and common measurement_start_time, probe_cc, probe_asn, and domain queries no longer require effectively full scans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, sql
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100