citusdata / citusdata/citus

Diagnostics counters

Open
#3,574 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Without looking at the specific workload of customers it is often hard to see where performance can be increased or where problems could be experienced. Postgres exposes some information for us already but there is little insight in citus diagnostics information.

As a simple example; our custom scan nodes read rows from the workers. It would be great if we could have some diagnostics counters around the rows it reads. Things I am thinking about would be:
- Number of rows (size of tuple store)
- Number of shards they came from
- Size in bytes (if easy to get from libpq) it read from the network
- Number of consumed rows, maybe we are consistently transferring too many rows

There could be countless metrics we would like to expose from a Citus perspective to get an insight in how the system performs and where we should optimise more.

One way of implementing such counters to limit the amount of resources spent on these counters could be as follows. We keep process local counters for the metrics citus is interested in. Either on process exit or a set time interval we could send these numbers of to postmaster (?) or an other background process where these numbers are combined from all postgres backends. Lastly we can make them available by some system view. Instead of inventing it our self we could also have a look at how `pg_stat_statement` implements the collection of its information into 1 location.

By exposing the metrics in a view an external monitoring or diagnostics collection system can swing by to extract this information from a running cluster and ship it to a central location for an operator that understands the metrics to look at.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.