citusdata / citusdata/citus

Active Sessions with ClientRead State

Open
#8,539 3 comments 0 reactions 0 assignees View on GitHub
documentation enhancement
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

### Observation

When running distributed queries (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `VACUUM`) on Citus clusters, `pg_stat_activity` frequently shows:
- `state='active'`
- `wait_event='ClientRead'`

This combination appears contradictory, as the Citus backend process isn't supposed to be active while waiting for client input.

### Potential Explanation

After investigating the Citus codebase, the behavior may be explained by the coordinator's distributed execution model: [src/backend/distributed/executor/adaptive_executor.c:1981](https://github.com/citusdata/citus/blob/4d6fb1d3a7cf4361cd15d4e652b3eeb3f4e48a55/src/backend/distributed/executor/adaptive_executor.c#L1981)

int eventCount =
WaitEventSetWait(execution->waitEventSet, timeout, execution->events,
execution->eventSetSize, WAIT_EVENT_CLIENT_READ);

### Hypothesis

The coordinator appears to:
1. Maintain active transaction contexts on itself and worker nodes
2. Wait for worker responses using `WAIT_EVENT_CLIENT_READ` (reusing PostgreSQL's wait event infrastructure)
3. This results in `state='active'` (transaction is open) combined with `wait_event='ClientRead'` (waiting to read from worker connections)

The same wait event is used across all distributed query types through the adaptive executor's main loop.

### Observation: Potential Enhancement Opportunity

From a monitoring and observability perspective, using `WAIT_EVENT_CLIENT_READ` for coordinator-worker communication may create some ambiguity:

- Standard interpretation: Backend waiting for the application client to send/consume data
- Citus reality: Coordinator waiting for worker nodes during normal distributed query execution

This could potentially make it difficult to distinguish between:
- Actual slow client issues (application problems)
- Normal distributed query coordination (expected Citus behavior)

### Question for consideration

Would it be valuable for Citus to use custom extension wait events (e.g., `"Citus:WorkerResponse"`) instead of `WAIT_EVENT_CLIENT_READ` for coordinator-worker communication?

This might improve observability and help operators distinguish between client-side issues and distributed query execution.

PostgreSQL's extension wait event framework (`WAIT_EVENT_EXTENSION`) could potentially support this without requiring core PostgreSQL changes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.