Active Sessions with ClientRead State
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
### Observation
When running distributed queries (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `VACUUM`) on Citus clusters, `pg_stat_activity` frequently shows:
- `state='active'`
- `wait_event='ClientRead'`
This combination appears contradictory, as the Citus backend process isn't supposed to be active while waiting for client input.
### Potential Explanation
After investigating the Citus codebase, the behavior may be explained by the coordinator's distributed execution model: [src/backend/distributed/executor/adaptive_executor.c:1981](https://github.com/citusdata/citus/blob/4d6fb1d3a7cf4361cd15d4e652b3eeb3f4e48a55/src/backend/distributed/executor/adaptive_executor.c#L1981)
int eventCount =
WaitEventSetWait(execution->waitEventSet, timeout, execution->events,
execution->eventSetSize, WAIT_EVENT_CLIENT_READ);
### Hypothesis
The coordinator appears to:
1. Maintain active transaction contexts on itself and worker nodes
2. Wait for worker responses using `WAIT_EVENT_CLIENT_READ` (reusing PostgreSQL's wait event infrastructure)
3. This results in `state='active'` (transaction is open) combined with `wait_event='ClientRead'` (waiting to read from worker connections)
The same wait event is used across all distributed query types through the adaptive executor's main loop.
### Observation: Potential Enhancement Opportunity
From a monitoring and observability perspective, using `WAIT_EVENT_CLIENT_READ` for coordinator-worker communication may create some ambiguity:
- Standard interpretation: Backend waiting for the application client to send/consume data
- Citus reality: Coordinator waiting for worker nodes during normal distributed query execution
This could potentially make it difficult to distinguish between:
- Actual slow client issues (application problems)
- Normal distributed query coordination (expected Citus behavior)
### Question for consideration
Would it be valuable for Citus to use custom extension wait events (e.g., `"Citus:WorkerResponse"`) instead of `WAIT_EVENT_CLIENT_READ` for coordinator-worker communication?
This might improve observability and help operators distinguish between client-side issues and distributed query execution.
PostgreSQL's extension wait event framework (`WAIT_EVENT_EXTENSION`) could potentially support this without requiring core PostgreSQL changes.
Contributor guide
Assessment
This issue has not been assessed yet.