oxidecomputer / oxidecomputer/omicron
Region replacement sagas apparently more prone to failed database connection claims
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
The issue was described to some extent in #8334 as the presence of many TLS certs exacerbated the saga failure mode. However, in environments where there is a relatively small number of TLS certs, we still observe errors associated with unavailable database claims.
On rack2 (with 12 sleds, i.e. >100 crucible zones), I used a dtrace scripts that captured database request latencies during sled expungement. The sled being expunged had 60 regions running on it that needed replacement. One of the queries turned out to be expensive and took up to 200+ seconds to complete. Here are the top 30 slowest queries from one of the three nexus zone db traces:
230870860 us daf6496a-9252-4007-b7c6-c45eb022c71d 'WITH...
215086310 us 0ebe903c-23b3-40b2-9b81-89f58c15d5f3 'WITH...
181667857 us ac59a370-0808-43c3-a444-606418d98949 'WITH...
179322072 us 6a859830-ce4d-41ae-9cee-466df4f6a0e6 'WITH...
156461931 us ddb29228-a776-440f-9314-834867b2e3ba 'WITH...
135466189 us 60bd183e-a224-4c8b-b743-891ddf3c0dc5 'WITH...
102951222 us a5b5bde5-461e-4359-88d3-1a7c17e59383 'WITH...
101351891 us 0a00b359-2dc2-4871-958c-289f06f5153b ''
64175405 us 246f13a2-f2f8-474a-84bd-481e5a92e123 'WITH...
63170638 us 45d2d8b6-a456-4925-8ea7-bd25f179ee6c 'WITH...
55098858 us 182fd83d-59f0-4a93-b2b0-1000be12e7bd 'WITH...
44932252 us 18a41292-fde0-4825-89fd-45c19a2d65eb 'WITH...
37586453 us 6a859830-ce4d-41ae-9cee-466df4f6a0e6 'WITH...
27900939 us 0a00b359-2dc2-4871-958c-289f06f5153b 'WITH...
13719127 us 3130c181-0468-4c21-bf1a-f4a36281c8d9 ''
12129699 us 246f13a2-f2f8-474a-84bd-481e5a92e123 'SELECT "crucible_dataset"."id"...
11110219 us 18a41292-fde0-4825-89fd-45c19a2d65eb 'SELECT "crucible_dataset"."id"...
9782805 us 182fd83d-59f0-4a93-b2b0-1000be12e7bd 'SELECT "crucible_dataset"."id"...
9749506 us 246f13a2-f2f8-474a-84bd-481e5a92e123 'SELECT "crucible_dataset"."id"...
9464062 us 182fd83d-59f0-4a93-b2b0-1000be12e7bd 'SELECT "disk"."id"...
9091935 us 45d2d8b6-a456-4925-8ea7-bd25f179ee6c 'SELECT "disk"."id"...
8538076 us ca08db09-7f59-4bcf-8345-d362bba026cb 'SELECT "crucible_dataset"."id"...
8491191 us e21b7ce9-f2ca-4117-8447-931265365d82 'SELECT "crucible_dataset"."id"...
8276285 us 6a859830-ce4d-41ae-9cee-466df4f6a0e6 'WITH...
7760126 us 246f13a2-f2f8-474a-84bd-481e5a92e123 'SELECT "region"."id"...
7641596 us 0a00b359-2dc2-4871-958c-289f06f5153b 'SELECT "crucible_dataset"."id"...
7573140 us 3130c181-0468-4c21-bf1a-f4a36281c8d9 'SELECT "disk"."id"...
7251640 us fefb4c63-637c-4905-8380-52efe23b3e6a 'SELECT "region"."id"...
7143919 us 3130c181-0468-4c21-bf1a-f4a36281c8d9 'SELECT "disk"."id"...
6906137 us 0a65d1dd-7fa8-4e73-8c3e-ea664e0c1ec1 'SELECT "disk"."id"...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or entry points are named. Start by reproducing sled expungement with many region replacements and inspect the slow database queries captured by the dtrace script. Done means identifying why the expensive queries cause unavailable database claims during region replacement and confirming that the failure mode is prevented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100