oxidecomputer / oxidecomputer/omicron

Populate ClickHouse timeseries schema from TOML definitions

Open
#5,942 0 comments 0 reactions 1 assignee View on GitHub

@bnaecker is already working on this.

Since Jun 24, 2024.

Metrics oximeter
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

As part of #5941, we'll move timeseries schema from ClickHouse into CockroachDB as the primary source of truth. ClickHouse will still need these definitions, however, to support querying and as a self-contained record of how to understand all the data in the database. This issue tracks populating the schema in ClickHouse from the underlying TOML definitions, rather than deriving those from the samples oximeter receives.

I'm not entirely sure how this will work yet, but as a strawman:

  • Add an API to oximeter for setting the schema
  • oximeter will insert (or update) these in ClickHouse. It should be the case that it can simply insert or update them all, since the TOML definitions will have been checked both at compile time and by Nexus when inserting into CockroachDB. It's not clear how to express that in ClickHouse yet, given the rather confusing ways modifications can be made.
  • oximeter should probably drop its internal schema cache. This is an optimization, and has also led to some bugs. It should just read the schema from ClickHouse whenever it needs them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.