oxidecomputer / oxidecomputer/omicron
Make single-node optional for oximeter (stage 3 of rolling out the replicated cluster)
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
On the dublin racklet which has v15rc1 (bc0fded), I made a direct switch from a single-node ClickHouse setup to the clustered one:
root@oxz_switch0:~# omdb nexus clickhouse-policy set cluster-only -w
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::21]:12221
Successfully inserted new policy at version 1
root@oxz_switch0:~# omdb -w nexus blueprints regenerate
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::21]:12221
generated new blueprint 9786ffd2-3d43-402c-84e2-c5dd4b1d87f6
(diff and target set the blueprint - excluding the console output to keep the bug descr brief)
root@oxz_switch0:~# omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::21]:12221
task: "blueprint_executor"
configured period: every 1m
currently executing: no
last completed activation: iter 60, triggered by a periodic timer firing
started at 2025-06-08T23:00:06.888Z (39s ago) and ran for 3675ms
target blueprint: 9786ffd2-3d43-402c-84e2-c5dd4b1d87f6
execution: enabled
status: completed (14 steps)
error: (none)
Afterwards, regenerated another blueprint which completed the single-node clickhouse expungement:
...
MODIFIED SLEDS:
sled 93a568b6-8568-441c-8efe-3d857b42aea3 (active, config generation 7):
physical disks:
-------------------------------------------------
vendor model serial disposition
-------------------------------------------------
1b96 WUS4C6432DSP3X3 A079DDCD in service
1b96 WUS4C6432DSP3X3 A079DF2B in service
1b96 WUS4C6432DSP3X3 A079DF42 in service
1b96 WUS4C6432DSP3X3 A079DF5A in service
1b96 WUS4C6432DSP3X3 A079DF5C in service
1b96 WUS4C6432DSP3X3 A079DFFB in service
1b96 WUS4C6432DSP3X3 A079E3AC in service
1b96 WUS4C6432DSP3X3 A079E3D6 in service
1b96 WUS4C6432DSP3X3 A079E435 in service
1b96 WUS4C6432DSP3X3 A079E4A7 in service
datasets:
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name dataset id disposition quota reservation compression
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/cockroachdb e39ee2f6-1bdd-492c-a0b6-b33caf0e7c72 in service none none off
oxp_b2ad458e-5d7c-4a73-a473-e8788d1d3c8b/crypt/cockroachdb 0a126f49-ffc3-4f5c-a523-a62e0290cef7 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crucible 64293700-d270-477f-86c4-7cb3d249b1db in service none none off
oxp_171af1ac-a157-401f-b5be-3f0302be051b/crucible 9bb0ddde-e905-41bd-9124-f7760f2e948c in service none none off
oxp_179376c9-9f9a-47c8-8f33-10ed3ad73d5f/crucible 80342c5c-e4dc-4de1-845c-b52927b96113 in service none none off
oxp_2655e3aa-9c0c-4ac5-ba6e-252e925c27f7/crucible eb34df00-4312-469a-8607-a8741cfecf46 in service none none off
oxp_3422a5fa-87ac-4a0b-a7d4-8f6dc2302f5c/crucible 04a1a454-0c7a-44d9-8eca-2e137e95bd85 in service none none off
oxp_363df070-9a74-46b1-9486-5001d2f46266/crucible cc10bc62-ddd1-4125-a31c-1b1e79ac9691 in service none none off
oxp_8574bffa-258e-4339-a887-a668913c2b2f/crucible 20a2009c-0bd0-4369-af71-deab6af58df5 in service none none off
oxp_b2ad458e-5d7c-4a73-a473-e8788d1d3c8b/crucible e7658fa7-4bd6-465c-aed9-88fd8da781c6 in service none none off
oxp_ce6a44da-6670-4666-8f21-bcdb5b3c6018/crucible c320b81b-76d3-45e8-9975-d11b04cff1bc in service none none off
oxp_f0470b51-5a7d-4cac-88c2-28bcb8743eb6/crucible a4133a4c-fde4-4ba7-b658-36b9b661db10 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/clickhouse 560e79a2-7b10-4518-b5f1-0b7def32d569 expunged none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/clickhouse_keeper ea0a9371-06ab-4a34-b7ab-472f65873dd1 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/clickhouse_server 12f957a6-cc43-4f0f-94ac-0f9e80173afd in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/internal_dns f5e2446b-c868-42e5-a992-1629897b0d69 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone 020484aa-9ecd-4c73-8537-48b68b1c5416 in service none none off
oxp_171af1ac-a157-401f-b5be-3f0302be051b/crypt/zone 8f406ce9-8a50-4682-8a00-4d47beb09aa5 in service none none off
oxp_179376c9-9f9a-47c8-8f33-10ed3ad73d5f/crypt/zone 16f3b17e-4713-4a63-8e92-166880f9ada3 in service none none off
oxp_2655e3aa-9c0c-4ac5-ba6e-252e925c27f7/crypt/zone e07782ae-bb8d-4a07-b7f5-e00d92863f49 in service none none off
oxp_3422a5fa-87ac-4a0b-a7d4-8f6dc2302f5c/crypt/zone e24408dc-4669-4222-a9b9-419597ab22f1 in service none none off
oxp_363df070-9a74-46b1-9486-5001d2f46266/crypt/zone 4e40b52f-001c-4342-8bc1-c84c69f129f3 in service none none off
oxp_8574bffa-258e-4339-a887-a668913c2b2f/crypt/zone a015ab09-ccff-41ea-8dc9-e4d0598a1c98 in service none none off
oxp_b2ad458e-5d7c-4a73-a473-e8788d1d3c8b/crypt/zone 1d891d18-9474-4d50-92a4-988eb48dd38a in service none none off
oxp_ce6a44da-6670-4666-8f21-bcdb5b3c6018/crypt/zone 8d073c2a-3425-4fa5-8f53-5a48718a0faf in service none none off
oxp_f0470b51-5a7d-4cac-88c2-28bcb8743eb6/crypt/zone cfe69869-91d2-4149-aeaa-ec76834e02c9 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone/oxz_clickhouse_21963d52-8c5f-41e0-9dc9-9edc2b4e6e83 c5dc8e21-87f0-4165-a010-239c47f4699f expunged none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone/oxz_clickhouse_keeper_bc387e68-2116-4684-bb1b-d98c0d36a9e7 aea5f322-912e-40b8-a058-b911f72e6417 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone/oxz_clickhouse_server_c8920696-713c-4957-827b-e774f07dac3b 92dcb9b9-2592-44d2-a796-dcb4d39399b3 in service none none off
oxp_b2ad458e-5d7c-4a73-a473-e8788d1d3c8b/crypt/zone/oxz_cockroachdb_515c8a5f-7d0e-4ed5-89d5-019dc6c91492 58505425-19a4-45e1-a8be-87f39994b88c in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone/oxz_cockroachdb_5a887587-c9b3-4a4b-919c-6966264c9b8c 3344d193-e5df-452e-8924-ed24f0613e15 in service none none off
oxp_ce6a44da-6670-4666-8f21-bcdb5b3c6018/crypt/zone/oxz_crucible_1090cec6-871f-46f3-8ce2-4f3d3b6cf4c2 17050eaf-4380-4f93-bbd1-9fd4ffb23292 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone/oxz_crucible_1c738fd3-450f-4585-9a5c-d7613bc8e7ea bddfcc81-ea2e-4bd3-a498-7972248d9565 in service none none off
oxp_171af1ac-a157-401f-b5be-3f0302be051b/crypt/zone/oxz_crucible_6d3967ec-63d1-4d08-8787-f728c49c478f 0e95985f-1705-4e10-ad70-ed7d53a6cbec in service none none off
oxp_8574bffa-258e-4339-a887-a668913c2b2f/crypt/zone/oxz_crucible_7048afc6-234c-4594-bc12-19c56eec39a6 f607dcb9-0c38-472a-b0b0-6c927ca6a976 in service none none off
oxp_179376c9-9f9a-47c8-8f33-10ed3ad73d5f/crypt/zone/oxz_crucible_76573a2c-ec26-4ba6-9dcc-dfb8fff17b67 b90860a7-4daf-4d9b-a18b-1e305c4283fd in service none none off
oxp_2655e3aa-9c0c-4ac5-ba6e-252e925c27f7/crypt/zone/oxz_crucible_c92a8e99-15e4-4e66-b731-a8eacefa7a72 ee241614-dccf-4b11-a1de-fea642abba29 in service none none off
oxp_f0470b51-5a7d-4cac-88c2-28bcb8743eb6/crypt/zone/oxz_crucible_e5385304-a261-4948-8ba6-378728e5764b 153e55e3-3cb6-4211-9366-456e1e3277a0 in service none none off
oxp_363df070-9a74-46b1-9486-5001d2f46266/crypt/zone/oxz_crucible_e53eb37d-465c-49d4-87a4-7d8e0e62fe1c 09cbc72a-f1d7-4436-a358-d4f3bfd785b7 in service none none off
oxp_3422a5fa-87ac-4a0b-a7d4-8f6dc2302f5c/crypt/zone/oxz_crucible_fab1cf1f-2438-42f5-81c5-72399de8caec 1210a425-bf60-4269-a6dc-8c53e5f050d7 in service none none off
oxp_b2ad458e-5d7c-4a73-a473-e8788d1d3c8b/crypt/zone/oxz_crucible_fd850920-5d7b-43ec-a18b-e66d837cdbfa a63769ed-d69f-4f0c-a195-c98aad4f3cb1 in service none none off
oxp_3422a5fa-87ac-4a0b-a7d4-8f6dc2302f5c/crypt/zone/oxz_crucible_pantry_f2f766e9-829e-4c41-8a99-e9c913595823 71631bbd-c70d-4269-9191-d1dc6e66f9dd in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/zone/oxz_internal_dns_641d39ca-8a04-4de7-96d6-10d9350c8816 e6a23917-186c-4784-ad5c-0fd9a2587338 in service none none off
oxp_8574bffa-258e-4339-a887-a668913c2b2f/crypt/zone/oxz_nexus_c0827889-a4dc-4172-9a6d-1230063991da 6472fe5c-818d-4b2e-bb59-7908a47faab0 in service none none off
oxp_171af1ac-a157-401f-b5be-3f0302be051b/crypt/zone/oxz_ntp_06fb0447-e175-4ec9-8b87-14abb1037ac5 32c74a11-e533-4f04-b4b3-19087449a188 in service none none off
oxp_13369df3-1ff6-48aa-b59d-43937d1fd2ac/crypt/debug 383c0495-5665-4a35-9dad-878aa53e71cb in service 100 GiB none gzip-9
oxp_171af1ac-a157-401f-b5be-3f0302be051b/crypt/debug 5c572780-1f39-4cfa-adbf-d48424fc93f4 in service 100 GiB none gzip-9
oxp_179376c9-9f9a-47c8-8f33-10ed3ad73d5f/crypt/debug 09487b24-72eb-4c59-a176-72bdc20b4173 in service 100 GiB none gzip-9
oxp_2655e3aa-9c0c-4ac5-ba6e-252e925c27f7/crypt/debug ef158c09-54d1-42f5-9894-f29e895f6b01 in service 100 GiB none gzip-9
oxp_3422a5fa-87ac-4a0b-a7d4-8f6dc2302f5c/crypt/debug 13b73bd6-4c46-4ffa-9ce6-f4b4a0729882 in service 100 GiB none gzip-9
oxp_363df070-9a74-46b1-9486-5001d2f46266/crypt/debug 4576a6c8-4ddd-4729-b3e0-a5d4fcde10f7 in service 100 GiB none gzip-9
oxp_8574bffa-258e-4339-a887-a668913c2b2f/crypt/debug 15376ebe-7c2a-471b-bd1a-02c95b4c57da in service 100 GiB none gzip-9
oxp_b2ad458e-5d7c-4a73-a473-e8788d1d3c8b/crypt/debug 09f64420-f8e5-46d6-8b0d-46eaedbc8c46 in service 100 GiB none gzip-9
oxp_ce6a44da-6670-4666-8f21-bcdb5b3c6018/crypt/debug 541a1a9b-dac3-432f-a1b9-7967fa64ddb7 in service 100 GiB none gzip-9
oxp_f0470b51-5a7d-4cac-88c2-28bcb8743eb6/crypt/debug bf486656-3a2d-4df8-90d8-0f9a6c019918 in service 100 GiB none gzip-9
omicron zones:
--------------------------------------------------------------------------------------------------------------------
zone type zone id image source disposition underlay IP
--------------------------------------------------------------------------------------------------------------------
clickhouse_keeper bc387e68-2116-4684-bb1b-d98c0d36a9e7 install dataset in service fd00:1122:3344:104::22
clickhouse_server c8920696-713c-4957-827b-e774f07dac3b install dataset in service fd00:1122:3344:104::23
cockroach_db 515c8a5f-7d0e-4ed5-89d5-019dc6c91492 install dataset in service fd00:1122:3344:104::3
cockroach_db 5a887587-c9b3-4a4b-919c-6966264c9b8c install dataset in service fd00:1122:3344:104::4
crucible 1090cec6-871f-46f3-8ce2-4f3d3b6cf4c2 install dataset in service fd00:1122:3344:104::d
crucible 1c738fd3-450f-4585-9a5c-d7613bc8e7ea install dataset in service fd00:1122:3344:104::8
crucible 6d3967ec-63d1-4d08-8787-f728c49c478f install dataset in service fd00:1122:3344:104::c
crucible 7048afc6-234c-4594-bc12-19c56eec39a6 install dataset in service fd00:1122:3344:104::f
crucible 76573a2c-ec26-4ba6-9dcc-dfb8fff17b67 install dataset in service fd00:1122:3344:104::10
crucible c92a8e99-15e4-4e66-b731-a8eacefa7a72 install dataset in service fd00:1122:3344:104::9
crucible e5385304-a261-4948-8ba6-378728e5764b install dataset in service fd00:1122:3344:104::a
crucible e53eb37d-465c-49d4-87a4-7d8e0e62fe1c install dataset in service fd00:1122:3344:104::b
crucible fab1cf1f-2438-42f5-81c5-72399de8caec install dataset in service fd00:1122:3344:104::e
crucible fd850920-5d7b-43ec-a18b-e66d837cdbfa install dataset in service fd00:1122:3344:104::7
crucible_pantry f2f766e9-829e-4c41-8a99-e9c913595823 install dataset in service fd00:1122:3344:104::6
internal_dns 641d39ca-8a04-4de7-96d6-10d9350c8816 install dataset in service fd00:1122:3344:2::1
internal_ntp 06fb0447-e175-4ec9-8b87-14abb1037ac5 install dataset in service fd00:1122:3344:104::11
nexus c0827889-a4dc-4172-9a6d-1230063991da install dataset in service fd00:1122:3344:104::5
* clickhouse 21963d52-8c5f-41e0-9dc9-9edc2b4e6e83 install dataset - expunged ⏳ fd00:1122:3344:104::21
└─ + expunged ✓
COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: d4d87aa2ad877a4cc2fddd0573952362739110de (unchanged)
cluster.preserve_downgrade_option: "22.1" (unchanged)
METADATA:
* internal DNS version: 5 -> 6
external DNS version: 3 (unchanged)
OXIMETER SETTINGS:
generation: 1 (unchanged)
read from:: SingleNode (unchanged)
CLICKHOUSE CLUSTER CONFIG:
generation::::::::::::::::::::::::::::::::::::: 2 (unchanged)
max used server id::::::::::::::::::::::::::::: 3 (unchanged)
max used keeper id::::::::::::::::::::::::::::: 5 (unchanged)
cluster name::::::::::::::::::::::::::::::::::: oximeter_cluster (unchanged)
cluster secret::::::::::::::::::::::::::::::::: 9d139ea9-a254-45da-8ac8-d1122c645c8c (unchanged)
* highest seen keeper leader committed log index: 0 -> 5146
clickhouse keepers at generation 2:
------------------------------------------------
zone id keeper id
------------------------------------------------
a00c4c4d-b750-4d4f-a728-8d68c3cc15d4 1
a4c6155d-2bda-4efb-b259-a82b0959cc81 2
bc387e68-2116-4684-bb1b-d98c0d36a9e7 3
c5363b5a-72fa-4d17-8964-d30eb122125c 4
f1f0c6ff-c9c2-4e8a-af56-518bdbfdabb2 5
clickhouse servers at generation 2:
------------------------------------------------
zone id server id
------------------------------------------------
110f6c30-076e-4c00-b57b-22b7262e26f5 1
73e08538-c8f2-4d9e-8bef-745f1e70d7c8 2
c8920696-713c-4957-827b-e774f07dac3b 3
root@oxz_switch0:~# omdb -w nexus blueprints target set 5f2148c1-c05f-4e6d-97b7-a4659f580ab5 enabled
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::21]:12221
set target blueprint to 5f2148c1-c05f-4e6d-97b7-a4659f580ab5
root@oxz_switch0:~# omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:103::4]:12221
task: "blueprint_executor"
configured period: every 1m
currently executing: iter 1122, triggered by a periodic timer firing
started at 2025-06-08T23:03:21.796Z, running for 2257ms
last completed activation: iter 1121, triggered by a periodic timer firing
started at 2025-06-08T23:02:21.792Z (62s ago) and ran for 2912ms
target blueprint: 5f2148c1-c05f-4e6d-97b7-a4659f580ab5
execution: enabled
status: completed (14 steps)
error: (none)
Then, switched oximeter to use the cluster, regenerated another blueprint:
root@oxz_switch0:~# omdb nexus oximeter-read-policy set cluster -w
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::21]:12221
Successfully inserted new policy at version 2
root@oxz_switch0:~# omdb -w nexus blueprints regenerate
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:101::21]:12221
generated new blueprint 1ec6e173-4a54-4108-a71d-593fc9f5ea32
The diffs showed:
OXIMETER SETTINGS:
* generation: 1 -> 2
* read from:: SingleNode -> Cluster
CLICKHOUSE CLUSTER CONFIG:
generation::::::::::::::::::::::::::::::::::::: 2 (unchanged)
max used server id::::::::::::::::::::::::::::: 3 (unchanged)
max used keeper id::::::::::::::::::::::::::::: 5 (unchanged)
cluster name::::::::::::::::::::::::::::::::::: oximeter_cluster (unchanged)
cluster secret::::::::::::::::::::::::::::::::: 9d139ea9-a254-45da-8ac8-d1122c645c8c (unchanged)
* highest seen keeper leader committed log index: 5146 -> 8544
clickhouse keepers at generation 2:
------------------------------------------------
zone id keeper id
------------------------------------------------
a00c4c4d-b750-4d4f-a728-8d68c3cc15d4 1
a4c6155d-2bda-4efb-b259-a82b0959cc81 2
bc387e68-2116-4684-bb1b-d98c0d36a9e7 3
c5363b5a-72fa-4d17-8964-d30eb122125c 4
f1f0c6ff-c9c2-4e8a-af56-518bdbfdabb2 5
clickhouse servers at generation 2:
------------------------------------------------
zone id server id
------------------------------------------------
110f6c30-076e-4c00-b57b-22b7262e26f5 1
73e08538-c8f2-4d9e-8bef-745f1e70d7c8 2
c8920696-713c-4957-827b-e774f07dac3b 3
After the blueprint was enabled and executed, checked that oximeter-reader was pointing to the cluster's three servers:
root@oxz_switch0:~# host -t SRV _oximeter-reader._tcp.control-plane.oxide.internal fd00:1122:3344:1::1
Using domain server:
Name: fd00:1122:3344:1::1
Address: fd00:1122:3344:1::1#53
Aliases:
_oximeter-reader._tcp.control-plane.oxide.internal has SRV record 0 0 9000 110f6c30-076e-4c00-b57b-22b7262e26f5.host.control-plane.oxide.internal.
_oximeter-reader._tcp.control-plane.oxide.internal has SRV record 0 0 9000 73e08538-c8f2-4d9e-8bef-745f1e70d7c8.host.control-plane.oxide.internal.
_oximeter-reader._tcp.control-plane.oxide.internal has SRV record 0 0 9000 c8920696-713c-4957-827b-e774f07dac3b.host.control-plane.oxide.internal.
The cluster appeared to be up and running. I was able to query the schema and some tables:
oximeter_cluster_3 :) SELECT
fields_string.field_value as transaction_name,
fields_u32.field_value as attempt,
measurements_f64.datum as attempt_duration,
measurements_f64.timestamp
FROM measurements_f64
INNER JOIN fields_string ON fields_string.timeseries_key = measurements_f64.timeseries_key
INNER JOIN fields_u32 ON fields_u32.timeseries_key = measurements_f64.timeseries_key
WHERE measurements_f64.timeseries_name = 'database_transaction:retry_data'
ORDER BY measurements_f64.timestamp ASC;
SELECT
fields_string.field_value AS transaction_name,
fields_u32.field_value AS attempt,
measurements_f64.datum AS attempt_duration,
measurements_f64.timestamp
FROM measurements_f64
INNER JOIN fields_string ON fields_string.timeseries_key = measurements_f64.timeseries_key
INNER JOIN fields_u32 ON fields_u32.timeseries_key = measurements_f64.timeseries_key
WHERE measurements_f64.timeseries_name = 'database_transaction:retry_data'
ORDER BY measurements_f64.timestamp ASC
Query id: 9b177192-50c5-45af-93f0-d32b7bee27e9
┌─transaction_name────────────┬─attempt─┬─attempt_duration─┬────measurements_f64.timestamp─┐
│ inventory_delete_collection │ 1 │ 0.078017852 │ 2025-06-08 23:04:26.215812977 │
└─────────────────────────────┴─────────┴──────────────────┴───────────────────────────────┘
1 row in set. Elapsed: 0.025 sec. Processed 55.57 thousand rows, 1.30 MB (2.20 million rows/s., 51.45 MB/s.)
Peak memory usage: 5.93 MiB.
But I wasn't getting data from OxQL (e.g. utilization, disk and instance metrics). In the oximeter log, I saw warnings about no backends:
root@oxz_oximeter_abad6def:~# tail -f /var/svc/log/oxide-oximeter\:default.log | looker
23:44:31.916Z DEBG oximeter (oximeter-agent): inserted rows into table
collector_id = abad6def-f9fb-4bad-917d-a981fe89a218
collector_ip = fd00:1122:3344:101::5
id = 5ffc28b8-6d39-4842-bbf2-85ca621f74a5
n_rows = 8
table_name = fields_uuid
23:44:31.924Z DEBG oximeter (oximeter-agent): inserted rows into table
collector_id = abad6def-f9fb-4bad-917d-a981fe89a218
collector_ip = fd00:1122:3344:101::5
id = 5ffc28b8-6d39-4842-bbf2-85ca621f74a5
n_rows = 4
table_name = measurements_cumulativeu64
23:45:00.427Z WARN oximeter (oximeter-agent): failed to insert some results into metric DB
collector_id = abad6def-f9fb-4bad-917d-a981fe89a218
collector_ip = fd00:1122:3344:101::5
error = Failed to check out connection to database: No backends found for this service
file = oximeter/collector/src/results_sink.rs:92
It would appear that oximeter is still trying to use the single-node Clickhouse after switching over to the cluster?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the omdb nexus clickhouse-policy set command, then follow blueprint regeneration and the blueprint_executor background task. Determine how the single-node ClickHouse resources are selected and what configuration makes them optional; done is a successful clustered deployment without requiring the single-node setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, rust
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100