oxidecomputer / oxidecomputer/omicron

`oximeter` insertions could be tuned or even self-tuned

Open
#10,390 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

We've had a number of issues related to ClickHouse performance lately. I decided to look at how often oximeter is inserting data, since the CH documentation is pretty clear that fewer inserts is better (larger batches -> fewer data part files -> more efficient background merges). So how are we doing today?

root@oxz_oximeter_edde139e:~# dtrace -n 'clickhouse*:::sql-query-start { @ = count() }' -n 'tick-10s { printa(@); exit(0); }'
dtrace: description 'clickhouse*:::sql-query-start ' matched 1 probe
dtrace: description 'tick-10s ' matched 1 probe
CPU     ID                    FUNCTION:NAME
125  94268                        :tick-10s
              198


On dogfood, we're inserting about 20 times per second. That's not terrible, but it could probably be better.

Right now, we insert every 5 seconds or when we get 5k samples. We could probably reduce that maximum interval to 1s, but we could tune the batch size automatically so that we actually do insert on average N times per second. That'd work by keeping track internally of every time we insert data, and ensure that number divided by our uptime is approximately N. If the current estimate is higher, use a shorter timer next time; if it's higher, use a longer one.

There are probably better algorithms too, but none of them should be too hard to implement.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests. Start by locating the oximeter ClickHouse insertion path and its current 5-second or 5,000-sample batching; compare the observed insertion rate with the target rate N, then validate that the chosen tuning behavior reduces insert frequency while preserving data delivery.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.