ClickHouse / ClickHouse/ClickHouse
Segmentation fault in RemoteQueryExecutorReadContext async fiber resume during shutdown/timeout (STID: 4348-2010)
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
## Summary
Recurring `Segmentation fault` (SIGSEGV, `STID: 4348-2010`) in the async remote-read fiber, seen only in `Stress test (arm_msan)`. The crash is inside the boost fiber `resume()` invoked from a `suspend_callback()` in `RemoteQueryExecutorReadContext::Task::run`. The faulting address is not mapped to any object (fiber-stack region), which points at a lifetime/teardown race on the fiber context rather than a data bug in the query itself.
Opening this per @alexey-milovidov's request on #110073: the signature is unrelated to that PR (it predates the branch and hits master and many unrelated PRs).
## Signature
```
Received signal Segmentation fault (11)
Address: 0x... Access: . Address not mapped to object.
contrib/boost/boost/context/fiber_ucontext.hpp:513: boost::context::fiber::resume() &&
src/Common/Fiber.h:64: Fiber::RoutineImpl::SuspendCallback::operator()()
src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:86: DB::RemoteQueryExecutorReadContext::Task::run(...)
src/Common/AsyncTaskExecutor.cpp:89: DB::AsyncTaskExecutor::Routine::operator()(...)
contrib/boost/boost/context/fiber_ucontext.hpp:313: fiber_capture_record<...>::run()
contrib/boost/boost/context/fiber_ucontext.hpp:81: fiber_entry_func<...>(void*)
```
`Fiber.h:64` is `SuspendCallback::operator()` -> `impl = std::move(impl).resume();` (yielding the fiber back to its sink). `RemoteQueryExecutorReadContext.cpp:86` is the trailing `suspend_callback();` after reading a packet body.
## How to reproduce
Not reliably reproducible outside CI so far. Both crashing queries in the latest occurrence use `remote(...)` and finish/tear down under load:
```sql
SELECT any(total) AS total_distinct_avg
FROM (SELECT number, avgDistinct(number) OVER () AS total FROM remote('127.0.0.{1,2,3}', numbers_mt(100000)));
SELECT count(), toString(number) AS k
FROM remote('127.0.0.{1,2}', numbers(10))
WHERE number > (queryID() = initialQueryID())
GROUP BY GROUPING SETS ((k), (number + 1, k)) ORDER BY k
SETTINGS group_by_two_level_threshold=9;
```
Both hit the same fiber-`resume()` fault. The stress harness runs many concurrent queries with the RandomQueryKiller and ZK fault injection, so query cancellation / socket timeout / connection teardown are all firing concurrently with the async read fiber.
## Root-cause direction (not yet confirmed)
The unmapped faulting address is the fiber's own stack region. The fiber yields via `SuspendCallback` (holding a reference `Impl & impl` to the sink fiber). A crash resuming that sink means the sink fiber context / `FiberStack` has been moved-from or destroyed while a suspend is still in flight. Suspect interaction between `AsyncTaskExecutor::resume()` / `cancel()` / `restart()` / `destroyFiber()` (`AsyncTaskExecutor.cpp`) and the in-flight `suspend_callback` on the read path (cancel-during-suspend or restart-during-suspend). MSan-only visibility suggests a use-of-uninitialized / freed context surfaced by MSan instrumentation rather than a hard bug that always faults.
## Occurrences (CIDB, `Stress test (arm_msan)`)
11 hits over the last month, master + 9 unrelated PRs, none of which touch this code path:
| date (UTC) | ref |
|---|---|
| 2026-07-11 | PR #110073 |
| 2026-07-02 | PR #108932 |
| 2026-06-29 | PR #100173 |
| 2026-06-23 | PR #108205 |
| 2026-06-18 | PR #106028 |
| 2026-06-11 | PRs #107160 #107161 #107074 #107109 |
| 2026-06-10 | PR #99105, **master** |
## CI report
Latest: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=110073&sha=f75f6b4383cd713bb07ad345f879953ae276e50e&name_0=PR&name_1=Stress%20test%20%28arm_msan%29
### Version info
- Resolved by: #110152
- Merged into: `26.7.1.1235` (included in `26.7` and later)
Contributor guide
Research direction
Start with the stack references in src/QueryPipeline/RemoteQueryExecutorReadContext.cpp, src/Common/Fiber.h, and src/Common/AsyncTaskExecutor.cpp, then review the resolution recorded in #110152. Run the Stress test (arm_msan) with the provided remote() queries to verify whether the fiber resume crash still occurs; done means the failure no longer recurs during cancellation and teardown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100