oxidecomputer / oxidecomputer/omicron
test failed in CI: integration_tests::oximeter::test_oximeter_reregistration
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on pull request #10999:
https://github.com/oxidecomputer/omicron/pull/10999/checks?check_run_id=92147298813
Log showing the specific test failure:
Excerpt from the log showing the failure:
TRY 1 FAIL [ 18.875s] (─────────) omicron-nexus::test_all integration_tests::oximeter::test_oximeter_reregistration
7754 2026-08-04T22:35:43.410Z stdout ───
7755 2026-08-04T22:35:43.410Z
7756 2026-08-04T22:35:43.410Z running 1 test
7757 2026-08-04T22:35:43.410Z test integration_tests::oximeter::test_oximeter_reregistration ... FAILED
7758 2026-08-04T22:35:43.410Z
7759 2026-08-04T22:35:43.410Z failures:
7760 2026-08-04T22:35:43.410Z
7761 2026-08-04T22:35:43.410Z failures:
7762 2026-08-04T22:35:43.410Z integration_tests::oximeter::test_oximeter_reregistration
7763 2026-08-04T22:35:43.410Z
7764 2026-08-04T22:35:43.410Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 570 filtered out; finished in 18.38s
7765 2026-08-04T22:35:43.410Z
7766 2026-08-04T22:35:43.410Z stderr ───
7767 2026-08-04T22:35:43.410Z log file: /var/tmp/omicron_tmp/test_all-18dcd38081c5cce2-test_oximeter_reregistration.22030.0.log
7768 2026-08-04T22:35:43.410Z note: configured to log to "/var/tmp/omicron_tmp/test_all-18dcd38081c5cce2-test_oximeter_reregistration.22030.0.log"
7769 2026-08-04T22:35:43.410Z DB URL: postgresql://root@[::1]:42306/omicron?sslmode=disable
7770 2026-08-04T22:35:43.410Z DB address: [::1]:42306
7771 2026-08-04T22:35:43.410Z log file: /var/tmp/omicron_tmp/test_all-18dcd38081c5cce2-test_oximeter_reregistration.22030.2.log
7772 2026-08-04T22:35:43.410Z note: configured to log to "/var/tmp/omicron_tmp/test_all-18dcd38081c5cce2-test_oximeter_reregistration.22030.2.log"
7773 2026-08-04T22:35:43.410Z Dendrite address: [::1]:62280 (dpd backend: [::1]:33425)
7774 2026-08-04T22:35:43.410Z
7775 2026-08-04T22:35:43.410Z thread 'integration_tests::oximeter::test_oximeter_reregistration' (2) panicked at nexus/tests/integration_tests/oximeter.rs:282:5:
7776 2026-08-04T22:35:43.410Z assertion `left == right` failed
7777 2026-08-04T22:35:43.410Z left: 60940
7778 2026-08-04T22:35:43.410Z right: 33468
7779 2026-08-04T22:35:43.410Z stack backtrace:
7780 2026-08-04T22:35:43.410Z 0: __rustc::rust_begin_unwind
7781 2026-08-04T22:35:43.410Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/panicking.rs:689:5
7782 2026-08-04T22:35:43.410Z 1: core::panicking::panic_fmt
7783 2026-08-04T22:35:43.410Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/panicking.rs:80:14
7784 2026-08-04T22:35:43.410Z 2: core::panicking::assert_failed_inner
7785 2026-08-04T22:35:43.411Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/panicking.rs:439:17
7786 2026-08-04T22:35:43.411Z 3: core::panicking::assert_failed::<u16, u16>
7787 2026-08-04T22:35:43.411Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/panicking.rs:394:5
7788 2026-08-04T22:35:43.411Z 4: {async_block#0}
7789 2026-08-04T22:35:43.411Z at ./tests/integration_tests/oximeter.rs:282:5
7790 2026-08-04T22:35:43.411Z 5: poll<&mut dyn core::future::future::Future<Output=()>>
7791 2026-08-04T22:35:43.411Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/future/future.rs:133:9
7792 2026-08-04T22:35:43.411Z 6: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7793 2026-08-04T22:35:43.411Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/future/future.rs:133:9
7794 2026-08-04T22:35:43.411Z 7: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7795 2026-08-04T22:35:43.411Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:778:70
7796 2026-08-04T22:35:43.411Z 8: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7797 2026-08-04T22:35:43.411Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:167:5
7798 2026-08-04T22:35:43.411Z 9: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
7799 2026-08-04T22:35:43.411Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:133:5
7800 2026-08-04T22:35:43.416Z 10: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7801 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:778:25
7802 2026-08-04T22:35:43.416Z 11: <tokio::runtime::scheduler::current_thread::Context>::enter::<core::task::poll::Poll<()>, <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}::{closure#0}>
7803 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:451:19
7804 2026-08-04T22:35:43.416Z 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
7805 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:777:44
7806 2026-08-04T22:35:43.416Z 13: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}
7807 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:68
7808 2026-08-04T22:35:43.416Z 14: <tokio::runtime::context::scoped::Scoped<tokio::runtime::scheduler::Context>>::set::<<tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
7809 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/scoped.rs:40:9
7810 2026-08-04T22:35:43.416Z 15: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}
7811 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:38
7812 2026-08-04T22:35:43.416Z 16: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#9}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#9}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
7813 2026-08-04T22:35:43.416Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/thread/local.rs:462:12
7814 2026-08-04T22:35:43.416Z 17: <std::thread::local::LocalKey<tokio::runtime::context::Context>>::with::<tokio::runtime::context::set_scheduler<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>::{closure#0}, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>)>
7815 2026-08-04T22:35:43.416Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/std/src/thread/local.rs:426:20
7816 2026-08-04T22:35:43.416Z 18: tokio::runtime::context::set_scheduler::<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, core::option::Option<()>), <tokio::runtime::scheduler::current_thread::CoreGuard>::enter<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>::{closure#0}>
7817 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context.rs:181:17
7818 2026-08-04T22:35:43.416Z 19: <tokio::runtime::scheduler::current_thread::CoreGuard>::enter::<<tokio::runtime::scheduler::current_thread::CoreGuard>::block_on<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>::{closure#0}, core::option::Option<()>>
7819 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:865:27
7820 2026-08-04T22:35:43.416Z 20: <tokio::runtime::scheduler::current_thread::CoreGuard>::block_on::<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>>
7821 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:765:24
7822 2026-08-04T22:35:43.416Z 21: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7823 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:205:33
7824 2026-08-04T22:35:43.416Z 22: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::current_thread::CurrentThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>::{closure#0}, ()>
7825 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/runtime.rs:65:16
7826 2026-08-04T22:35:43.416Z 23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7827 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/current_thread/mod.rs:193:9
7828 2026-08-04T22:35:43.416Z 24: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = ()>>>
7829 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:371:52
7830 2026-08-04T22:35:43.416Z 25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
7831 2026-08-04T22:35:43.416Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:345:18
7832 2026-08-04T22:35:43.416Z 26: test_oximeter_reregistration
7833 2026-08-04T22:35:43.416Z at ./tests/integration_tests/oximeter.rs:344:29
7834 2026-08-04T22:35:43.417Z 27: test_all::integration_tests::oximeter::test_oximeter_reregistration::{closure#0}
7835 2026-08-04T22:35:43.417Z at ./tests/integration_tests/oximeter.rs:75:40
7836 2026-08-04T22:35:43.417Z 28: <test_all::integration_tests::oximeter::test_oximeter_reregistration::{closure#0} as core::ops::function::FnOnce<()>>::call_once
7837 2026-08-04T22:35:43.417Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/ops/function.rs:250:5
7838 2026-08-04T22:35:43.417Z 29: <fn() -> core::result::Result<(), alloc::string::String> as core::ops::function::FnOnce<()>>::call_once
7839 2026-08-04T22:35:43.417Z at /rustc/8bab26f4f68e0e26f0bb7960be334d5b520ea452/library/core/src/ops/function.rs:250:5
7840 2026-08-04T22:35:43.417Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
7841 2026-08-04T22:35:43.417Z WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7842 2026-08-04T22:35:43.417Z WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpckLQTD"
7843 2026-08-04T22:35:43.417Z If you would like to access the database for debugging, run the following:
7844 2026-08-04T22:35:43.417Z
7845 2026-08-04T22:35:43.417Z # Run the database
7846 2026-08-04T22:35:43.417Z cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpckLQTD/data"
7847 2026-08-04T22:35:43.417Z # Access the database. Note the port may change if you run multiple databases.
7848 2026-08-04T22:35:43.417Z cockroach sql --host=localhost:32221 --insecure
7849 2026-08-04T22:35:43.417Z WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 22098) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-18dcd38081c5cce2-test_oximeter_reregistration.22030.1-clickhouse-P4P7RE)
7850 2026-08-04T22:35:43.417Z failed to clean up ClickHouse data dir:
7851 2026-08-04T22:35:43.417Z - /var/tmp/omicron_tmp/test_all-18dcd38081c5cce2-test_oximeter_reregistration.22030.1-clickhouse-P4P7RE: File exists (os error 17)
7852 2026-08-04T22:35:43.417Z WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7853 2026-08-04T22:35:43.417Z WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpOw1U4U
7854 2026-08-04T22:35:43.417Z WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7855 2026-08-04T22:35:43.417Z WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpOcOzYI
7856 2026-08-04T22:35:43.417Z WARN: dropped DdmInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
7857 2026-08-04T22:35:43.417Z WARN: ddmd temporary directory leaked: /var/tmp/omicron_tmp/.tmpxnsytN
7858 2026-08-04T22:35:43.417Z
nextest tried rerunning it, and it passed, so this is definitely a flake.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with nexus/tests/integration_tests/oximeter.rs, especially integration_tests::oximeter::test_oximeter_reregistration and the assertion at line 282. Reproduce that integration test and inspect the CI log and PostgreSQL-backed test setup to determine why the values differ. Done means the test passes reliably in CI without weakening its assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- ci-cd, databases, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100