ClickHouse / ClickHouse/clickhouse-rs

Flaky test: opentelemetry::error_sets_span_status intermittently panics with "expected last span data"

Open
#463 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
559
Forks
172
Avg merge
1d 19h
Merged PRs (30d)
3

Description

### Describe the bug

`opentelemetry::error_sets_span_status` (`tests/it/opentelemetry.rs`) fails intermittently
in CI. It is not tied to a ClickHouse server version and not tied to a particular commit:
it has failed on `main` in scheduled runs on 4 separate days in the last week, on two
different server legs, and on an unrelated PR branch on a third leg. The same check passes
on the other legs of the same run.

The assertion that fails is the `expect` at `tests/it/opentelemetry.rs:149`:

```
thread 'opentelemetry::error_sets_span_status' panicked at tests/it/opentelemetry.rs:149:43:
expected last span data
```

so `LAST_SPAN_DATA` was `None` — no span reached the test's span processor on the test's
own thread. It is not the later status assertion (`expected error status, got ...`).

### Steps to reproduce

The failure is non-deterministic. It appears only in CI so far.

1. Run the `it` suite with the `opentelemetry` feature enabled — in CI this is the
`cargo llvm-cov test --workspace --no-report --all-features` step of `test-local`.
The three `opentelemetry` tests only compile under `--all-features`, so each CI leg
executes them once.
2. Repeat. Roughly 4 of the last 14 executions failed.

### Evidence of non-determinism

Failing runs (all `test-local`, all the same assertion, all on unrelated commits):

* 2026-08-24 scheduled, `main`, leg `head` — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32692955489/job/97329789872
* 2026-08-23 scheduled, `main`, leg `head` — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32619689568/job/97145842648
* 2026-08-22 scheduled, `main`, leg `head` — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32553579597/job/96984138868
* 2026-08-17 scheduled, `main`, leg `latest` — https://github.com/ClickHouse/clickhouse-rs/actions/runs/31997254612/job/95290996512
* 2026-08-24, PR run, leg pinned to server 26.6 — https://github.com/ClickHouse/clickhouse-rs/actions/runs/32693722038

Passing runs of the same check on the same code: the `latest` leg passed in each of the
three scheduled runs above where `head` failed, and the whole scheduled matrix was green on
2026-08-13, 08-14, 08-15, 08-16, 08-18, 08-19, 08-20 and 08-21.

Because the failures cover server legs `latest`, `head` and a pinned `26.6`, the failure is
not a server-version dependency. `tests/it/opentelemetry.rs` has not changed since May 2026,
so it is also not a recent change to the test itself.

### Local reproduction

Not reproduced on an idle or loaded developer box:

* `cargo test --all-features --test it opentelemetry::` — 0 failures in 25 consecutive runs.
* Full `it` suite, `--all-features`, 3 runs, under heavy CPU contention (4-core box, load
average ~30) — `error_sets_span_status` passed in all 3.

Server: ClickHouse 26.3 in Docker. Client: `main` at b508875. Rust 1.89.

### Additional observation

In the failing CI runs the captured output for the test is empty — the
`eprintln!("last span data: {span:?}")` inside `LastSpanProcessor::on_end`
(`tests/it/opentelemetry.rs:180`) produced nothing on that test's thread. `libtest` captures
output per thread, so on a failing run either no span was ended at all, or the span was
ended on a different thread than the one running the test body. On a passing run the same
test emits that line for both the `query` and the `response` span.

### Suspected fragile area (not a confirmed root cause)

The test observes span data through a **thread-local**:

```rust
thread_local! {
static LAST_SPAN_DATA: Cell> = const { Cell::new(None) };
}
```

`LastSpanProcessor::on_end` writes it, and the test body reads it with
`LAST_SPAN_DATA.take()`. The tracer provider is global and installed once per process
(`static ONCE: Once`), and the subscriber is installed per test with a thread-local
`set_default()`, while the three `opentelemetry` tests run concurrently on different
`libtest` threads. That makes the assertion depend on the span being ended on exactly the
thread that runs the test body, and on the span being ended before control returns to the
test body — neither of which the test enforces. We were not able to pin down which of these
actually breaks in CI, so this is offered as the area to look at, not as a diagnosis.

### Suggested direction

Make the observation independent of the thread the span ends on, and make the test wait for
the span instead of assuming it has already arrived — for example store exported spans in a
process-wide `Mutex>` keyed by trace id and look up the trace id the test
created, rather than "the last span on this thread". Forcing the `opentelemetry` tests onto
a single thread would also remove the cross-thread part, but not the ordering part.

Please do not paper over it with a retry or a sleep — if the span really can be ended on a
foreign thread or after the request future completes, that is worth knowing.

### Configuration
#### Environment
* Client version: `main` (b508875), also observed on every `main` commit in the window above
* OS: `ubuntu-latest` GitHub runner (CI); reproduction attempt on Ubuntu 24.04, 4 cores

#### ClickHouse server
* ClickHouse Server version: fails on `latest`, `head` and pinned `26.6`; local attempt on 26.3
* ClickHouse Server non-default settings, if any: none (repo `docker-compose.yml`)

---

Found by automated CI monitoring of our own PRs, then confirmed against the scheduled `main`
runs. Reported by @polyglotAI-bot.

Contributor guide

Open the contributing guide

Research direction

Start with tests/it/opentelemetry.rs, especially the assertion at line 149, LAST_SPAN_DATA, and LastSpanProcessor::on_end around line 180. Run the all-features integration test command and repeat it while tracing how spans are ended and observed across concurrent test threads. Done means error_sets_span_status reliably observes its own query and response spans in CI without retries or sleeps, then verifies the expected error status.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
observability, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.