Diagnostics counters
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Without looking at the specific workload of customers it is often hard to see where performance can be increased or where problems could be experienced. Postgres exposes some information for us already but there is little insight in citus diagnostics information.
As a simple example; our custom scan nodes read rows from the workers. It would be great if we could have some diagnostics counters around the rows it reads. Things I am thinking about would be:
- Number of rows (size of tuple store)
- Number of shards they came from
- Size in bytes (if easy to get from libpq) it read from the network
- Number of consumed rows, maybe we are consistently transferring too many rows
There could be countless metrics we would like to expose from a Citus perspective to get an insight in how the system performs and where we should optimise more.
One way of implementing such counters to limit the amount of resources spent on these counters could be as follows. We keep process local counters for the metrics citus is interested in. Either on process exit or a set time interval we could send these numbers of to postmaster (?) or an other background process where these numbers are combined from all postgres backends. Lastly we can make them available by some system view. Instead of inventing it our self we could also have a look at how `pg_stat_statement` implements the collection of its information into 1 location.
By exposing the metrics in a view an external monitoring or diagnostics collection system can swing by to extract this information from a running cluster and ship it to a central location for an operator that understands the metrics to look at.
Contributor guide
Assessment
This issue has not been assessed yet.