oxidecomputer / oxidecomputer/omicron
Populate ClickHouse timeseries schema from TOML definitions
Open
@bnaecker is already working on this.
Since Jun 24, 2024.
Metrics
oximeter
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
As part of #5941, we'll move timeseries schema from ClickHouse into CockroachDB as the primary source of truth. ClickHouse will still need these definitions, however, to support querying and as a self-contained record of how to understand all the data in the database. This issue tracks populating the schema in ClickHouse from the underlying TOML definitions, rather than deriving those from the samples oximeter receives.
I'm not entirely sure how this will work yet, but as a strawman:
- Add an API to
oximeterfor setting the schema oximeterwill insert (or update) these in ClickHouse. It should be the case that it can simply insert or update them all, since the TOML definitions will have been checked both at compile time and by Nexus when inserting into CockroachDB. It's not clear how to express that in ClickHouse yet, given the rather confusing ways modifications can be made.oximetershould probably drop its internal schema cache. This is an optimization, and has also led to some bugs. It should just read the schema from ClickHouse whenever it needs them.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.