oxidecomputer / oxidecomputer/omicron

test failure: `oximeter-db::integration_test test_cluster`

Open
#11,262 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

On branch ben/poison-clickhouse-connections-on-failure commit a83396a37, I saw this test failure:

bnaecker@flint : ~/file-cabinet/oxide/omicron $ cargo nextest run -p oximeter-db
info: experimental features enabled: setup-scripts, benchmarks
    Blocking waiting for file lock on build directory
   Compiling oximeter-db v0.1.0 (/Users/bnaecker/file-cabinet/oxide/omicron/oximeter/db)
   Compiling nexus-types v0.1.0 (/Users/bnaecker/file-cabinet/oxide/omicron/nexus/types)
   Compiling nexus-config v0.1.0 (/Users/bnaecker/file-cabinet/oxide/omicron/nexus-config)
   Compiling omicron-test-utils v0.1.0 (/Users/bnaecker/file-cabinet/oxide/omicron/test-utils)
   Compiling oximeter-test-utils v0.1.0 (/Users/bnaecker/file-cabinet/oxide/omicron/oximeter/test-utils)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10.35s
────────────
 Nextest run ID f80c4026-074d-483e-88f6-ba43855eaeb5 with nextest profile: default
    Starting 255 tests across 3 binaries
        PASS [   0.012s] oximeter-db client::oxql::tests::test_chunk_consistent_key_groups_split_middle_of_key_group
        PASS [   0.013s] oximeter-db client::oxql::tests::test_chunk_consistent_key_groups_split_key_group_multiple_times
...
       PASS [   1.355s] oximeter-db oxql::ast::grammar::tests::test_recognize_escape_sequence_with_valid_unicode_sequence
        FAIL [   7.456s] oximeter-db::integration_test test_cluster
  stdout ───

    running 1 test
    Deploying keeper: /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.1-oximeter-clickward-test/keeper-1
    Deploying keeper: /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.1-oximeter-clickward-test/keeper-3
    Deploying keeper: /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.1-oximeter-clickward-test/keeper-2
    Deploying clickhouse server: /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.1-oximeter-clickward-test/clickhouse-1
    Deploying clickhouse server: /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.1-oximeter-clickward-test/clickhouse-2
    test test_cluster ... FAILED

    failures:

    failures:
        test_cluster

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out; finished in 7.44s

  stderr ───
    log file: /var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.0.log
    note: configured to log to "/var/folders/67/2tlym22x1r3d2kwbh84j298w0000gn/T/integration_test-d5a1d0774d2cfe91-test_cluster.32777.0.log"

    thread 'test_cluster' (74469256) panicked at oximeter/db/tests/integration_test.rs:277:5:
    expected non-zero UserTimeMicroseconds in profile summaries, got 0
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This assertion was introduced in #10172. In that PR, @jmcarp noted that the profile event messages with the relevant value, UserTimeMicroseconds, is occassionally missing, in which case we default to returning 0 for it. That trips this assertion, by construction. We should probably either remove this assertion or return a map of Option<i64>s from profile_summary() to distinguish 0 from "missing".

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with oximeter/db/tests/integration_test.rs at the test_cluster assertion around line 277, then inspect profile_summary() and the profile event handling it uses. Reproduce the test with cargo nextest run -p oximeter-db, and make the behavior distinguish a missing UserTimeMicroseconds value from a real zero so test_cluster no longer fails for missing data.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, rust
Domain
databases, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.