oxidecomputer / oxidecomputer/omicron
Oximeter creates and destroys qorb pools while waiting for schema migration
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
When Oximeter starts up and clickhouse is on an old schema version, it goes into an infinite loop waiting for an operator to run schema migration: https://github.com/oxidecomputer/omicron/blob/2a9f1641e68410d3406d950eee9864df971a5afc/oximeter/collector/src/lib.rs#L278-L284
However, in every iteration of this loop, we create a new qorb pool and then drop it when returning the error indicating the version isn't what we expect: https://github.com/oxidecomputer/omicron/blob/2a9f1641e68410d3406d950eee9864df971a5afc/oximeter/collector/src/agent.rs#L704-L719
In combination with qorb versions prior to the fix in https://github.com/oxidecomputer/qorb/pull/78, this results in 8 leaked TCP connections for every retry attempt.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the schema-version retry loop in oximeter/collector/src/lib.rs around lines 278-284, then trace pool creation and error handling in oximeter/collector/src/agent.rs around lines 704-719. Reproduce or inspect the retry path with an old ClickHouse schema and verify that repeated attempts no longer leak TCP connections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100