ClickHouse / ClickHouse/ClickHouse

Segmentation fault in RemoteQueryExecutorReadContext async fiber resume during shutdown/timeout (STID: 4348-2010)

Open
#110,124 10 comments 0 reactions 0 assignees View on GitHub
comp-query-execution
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

## Summary

Recurring `Segmentation fault` (SIGSEGV, `STID: 4348-2010`) in the async remote-read fiber, seen only in `Stress test (arm_msan)`. The crash is inside the boost fiber `resume()` invoked from a `suspend_callback()` in `RemoteQueryExecutorReadContext::Task::run`. The faulting address is not mapped to any object (fiber-stack region), which points at a lifetime/teardown race on the fiber context rather than a data bug in the query itself.

Opening this per @alexey-milovidov's request on #110073: the signature is unrelated to that PR (it predates the branch and hits master and many unrelated PRs).

## Signature

```
Received signal Segmentation fault (11)
Address: 0x... Access: . Address not mapped to object.

contrib/boost/boost/context/fiber_ucontext.hpp:513: boost::context::fiber::resume() &&
src/Common/Fiber.h:64: Fiber::RoutineImpl::SuspendCallback::operator()()
src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:86: DB::RemoteQueryExecutorReadContext::Task::run(...)
src/Common/AsyncTaskExecutor.cpp:89: DB::AsyncTaskExecutor::Routine::operator()(...)
contrib/boost/boost/context/fiber_ucontext.hpp:313: fiber_capture_record<...>::run()
contrib/boost/boost/context/fiber_ucontext.hpp:81: fiber_entry_func<...>(void*)
```

`Fiber.h:64` is `SuspendCallback::operator()` -> `impl = std::move(impl).resume();` (yielding the fiber back to its sink). `RemoteQueryExecutorReadContext.cpp:86` is the trailing `suspend_callback();` after reading a packet body.

## How to reproduce

Not reliably reproducible outside CI so far. Both crashing queries in the latest occurrence use `remote(...)` and finish/tear down under load:

```sql
SELECT any(total) AS total_distinct_avg
FROM (SELECT number, avgDistinct(number) OVER () AS total FROM remote('127.0.0.{1,2,3}', numbers_mt(100000)));

SELECT count(), toString(number) AS k
FROM remote('127.0.0.{1,2}', numbers(10))
WHERE number > (queryID() = initialQueryID())
GROUP BY GROUPING SETS ((k), (number + 1, k)) ORDER BY k
SETTINGS group_by_two_level_threshold=9;
```

Both hit the same fiber-`resume()` fault. The stress harness runs many concurrent queries with the RandomQueryKiller and ZK fault injection, so query cancellation / socket timeout / connection teardown are all firing concurrently with the async read fiber.

## Root-cause direction (not yet confirmed)

The unmapped faulting address is the fiber's own stack region. The fiber yields via `SuspendCallback` (holding a reference `Impl & impl` to the sink fiber). A crash resuming that sink means the sink fiber context / `FiberStack` has been moved-from or destroyed while a suspend is still in flight. Suspect interaction between `AsyncTaskExecutor::resume()` / `cancel()` / `restart()` / `destroyFiber()` (`AsyncTaskExecutor.cpp`) and the in-flight `suspend_callback` on the read path (cancel-during-suspend or restart-during-suspend). MSan-only visibility suggests a use-of-uninitialized / freed context surfaced by MSan instrumentation rather than a hard bug that always faults.

## Occurrences (CIDB, `Stress test (arm_msan)`)

11 hits over the last month, master + 9 unrelated PRs, none of which touch this code path:

| date (UTC) | ref |
|---|---|
| 2026-07-11 | PR #110073 |
| 2026-07-02 | PR #108932 |
| 2026-06-29 | PR #100173 |
| 2026-06-23 | PR #108205 |
| 2026-06-18 | PR #106028 |
| 2026-06-11 | PRs #107160 #107161 #107074 #107109 |
| 2026-06-10 | PR #99105, **master** |

## CI report

Latest: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110073&sha=f75f6b4383cd713bb07ad345f879953ae276e50e&name_0=PR&name_1=Stress%20test%20%28arm_msan%29

### Version info
- Resolved by: #110152
- Merged into: `26.7.1.1235` (included in `26.7` and later)

Contributor guide

Open the contributing guide

Research direction

Start with the stack references in src/QueryPipeline/RemoteQueryExecutorReadContext.cpp, src/Common/Fiber.h, and src/Common/AsyncTaskExecutor.cpp, then review the resolution recorded in #110152. Run the Stress test (arm_msan) with the provided remote() queries to verify whether the fiber resume crash still occurs; done means the failure no longer recurs during cancellation and teardown.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.