oxidecomputer / oxidecomputer/omicron
`oximeter` insertions could be tuned or even self-tuned
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
We've had a number of issues related to ClickHouse performance lately. I decided to look at how often oximeter is inserting data, since the CH documentation is pretty clear that fewer inserts is better (larger batches -> fewer data part files -> more efficient background merges). So how are we doing today?
root@oxz_oximeter_edde139e:~# dtrace -n 'clickhouse*:::sql-query-start { @ = count() }' -n 'tick-10s { printa(@); exit(0); }'
dtrace: description 'clickhouse*:::sql-query-start ' matched 1 probe
dtrace: description 'tick-10s ' matched 1 probe
CPU ID FUNCTION:NAME
125 94268 :tick-10s
198
On dogfood, we're inserting about 20 times per second. That's not terrible, but it could probably be better.
Right now, we insert every 5 seconds or when we get 5k samples. We could probably reduce that maximum interval to 1s, but we could tune the batch size automatically so that we actually do insert on average N times per second. That'd work by keeping track internally of every time we insert data, and ensure that number divided by our uptime is approximately N. If the current estimate is higher, use a shorter timer next time; if it's higher, use a longer one.
There are probably better algorithms too, but none of them should be too hard to implement.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests. Start by locating the oximeter ClickHouse insertion path and its current 5-second or 5,000-sample batching; compare the observed insertion rate with the target rate N, then validate that the chosen tuning behavior reduces insert frequency while preserving data delivery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100