config: expose custom metrics

Product: Tarantool

Since: 3.8

Related dev. issue(s):

• https://github.com/tarantool/tarantool/pull/12708
• https://github.com/tarantool/metrics/pull/543

Root document:

• https://www.tarantool.io/en/doc/latest/admin/monitoring/getting_started/#configuring-metrics
• https://www.tarantool.io/en/doc/latest/admin/monitoring/getting_started/#creating-custom-metrics
• https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/#metrics
• https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/#metrics-include
• https://www.tarantool.io/en/doc/latest/reference/configuration/configuration_reference/#metrics-exclude
• https://www.tarantool.io/en/doc/latest/reference/reference_lua/metrics/#metrics-cfg
• https://www.tarantool.io/en/doc/latest/reference/reference_lua/metrics/#custom-metrics

SME: @mandesero

Summary

Document selector-based filtering for custom metrics.

The metrics.include and metrics.exclude configuration options now support not only built-in metric groups, such as cpu, network, info, vinyl, and all, but also custom metric selectors.

This makes custom metrics configurable through the same declarative configuration mechanism as built-in metrics.

Before this change, an application or role could register custom metrics using the metrics module, but these metrics could not be selectively enabled or disabled through the metrics.include / metrics.exclude options in the Tarantool configuration.

Now, application code, roles, and third-party modules can assign hierarchical selectors to custom metrics. These selectors can then be used in the Tarantool configuration.

Concepts to document

Custom metric selector

A custom metric selector is a hierarchical string assigned to a custom metric collector or callback.

A selector can be used to group related custom metrics and control their visibility in the metrics output.

Recommended selector naming patterns:

• roles.<role-name> for metrics provided by Tarantool roles.
• roles.<role-name>.<metric-name> for a specific metric of a role.
• app.<module-name> for application-level metrics.
• app.<module-name>.<metric-name> for a specific application metric.

Examples:

roles.crud-router
roles.crud-router.crud_requests
roles.crud-router.crud_errors
roles.tqe-storage
app.queue
app.queue.jobs_ready

Selectors are hierarchical. A selector matches either exactly or by dot prefix.

For example, the selector roles.crud-router matches all of the following:

roles.crud-router
roles.crud-router.crud_requests
roles.crud-router.crud_errors
roles.crud-router.request_latency

But it does not match:

roles.crud-storage
roles.crud-router-v2
app.crud-router

Configuring custom metrics

Add a new subsection to the monitoring guide near “Configuring metrics” / “Creating custom metrics”.

Suggested text:

Custom metrics can be enabled or disabled using the metrics.include and metrics.exclude options in the Tarantool configuration.

To make a custom metric configurable, assign it a selector. A selector is a hierarchical string used for filtering custom metric collectors and callbacks.

Example:

local metrics = require('metrics')

local crud_requests = metrics.gauge(
    'crud_requests',
    'Number of CRUD requests',
    {selector = 'roles.crud-router.crud_requests'}
)

local crud_errors = metrics.gauge(
    'crud_errors',
    'Number of CRUD request errors',
    {selector = 'roles.crud-router.crud_errors'}
)

crud_requests:set(10)
crud_errors:set(1)

Then enable or disable these metrics in the Tarantool configuration:

metrics:
  include:
    - cpu
    - roles.crud-router
  exclude:
    - roles.crud-router.crud_errors

This configuration:

• enables the built-in cpu metric group;
• enables custom metrics whose selectors are equal to roles.crud-router or start with roles.crud-router.;
• disables the roles.crud-router.crud_errors custom metric.

The metrics.exclude option has higher priority than metrics.include. This means that you can enable a whole selector subtree and then disable a specific metric or nested group.

Using namespaces

Add this subsection to the custom metrics guide or Lua metrics reference.

For convenience, use metrics.namespace() to create a namespace for related custom metrics.

A namespace automatically assigns hierarchical selectors to collectors registered through it.

Example:

local metrics = require('metrics')

local crud = metrics.namespace('roles.crud-router')

local requests = crud:gauge(
    'crud_requests',
    'Number of CRUD requests'
)

local errors = crud:gauge(
    'crud_errors',
    'Number of CRUD request errors'
)

requests:set(10)
errors:set(1)

The metric names are:

crud_requests
crud_errors

The corresponding selectors are:

roles.crud-router.crud_requests
roles.crud-router.crud_errors

These selectors can be used in the Tarantool configuration:

metrics:
  include:
    - roles.crud-router
  exclude:
    - roles.crud-router.crud_errors

Using selectors with callback-based metrics

Custom metrics can also be updated in callbacks.

Example with an explicit selector:

local metrics = require('metrics')

local in_flight = metrics.gauge(
    'crud_requests_in_flight',
    'Number of in-flight CRUD requests',
    {selector = 'roles.crud-router.in_flight'}
)

metrics.register_callback(function()
    in_flight:set(get_in_flight_requests())
end, {selector = 'roles.crud-router'})

In this example, the callback is associated with the roles.crud-router selector. The metric itself is associated with the roles.crud-router.in_flight selector.

The metric can be enabled using either selector:

metrics:
  include:
    - roles.crud-router

Or more specifically:

metrics:
  include:
    - roles.crud-router.in_flight

Example with a namespace:

local metrics = require('metrics')

local crud = metrics.namespace('roles.crud-router')

local in_flight = crud:gauge(
    'in_flight',
    'Number of in-flight CRUD requests'
)

crud:register_callback(function()
    in_flight:set(get_in_flight_requests())
end)

The namespace assigns the roles.crud-router selector to the callback and the roles.crud-router.in_flight selector to the metric.

Lua API reference updates

Update metrics.cfg()

Current documentation says that cfg.include and cfg.exclude contain default metric names. Extend it to mention custom selectors.

Suggested replacement / addition:

cfg.include can contain:

• all to enable all supported built-in metrics;
• none to disable all built-in metrics;
• names of built-in metric groups, such as cpu, network, info, or vinyl;
• custom metric selectors.

cfg.exclude can contain:

• names of built-in metric groups;
• custom metric selectors.

cfg.exclude has higher priority than cfg.include.

Unknown strings in cfg.include and cfg.exclude are treated as custom metric selectors.

Example:

local metrics = require('metrics')

metrics.cfg{
    include = {'info', 'cpu', 'roles.crud-router'},
    exclude = {'roles.crud-router.crud_errors'},
}

This configuration enables:

• the built-in info metric group;
• the built-in cpu metric group;
• custom metrics under the roles.crud-router selector.

It disables the custom metric with the roles.crud-router.crud_errors selector.

Add metrics.namespace(selector)

metrics.namespace(selector)

Create a namespace object for registering related custom metrics.

Parameters:

• selector (string) -- selector prefix for all collectors and callbacks registered through this namespace.

Return value:

• A namespace object.

The namespace object provides collector creation methods similar to the metrics module:

• namespace:counter(name[, help, metainfo])
• namespace:gauge(name[, help, metainfo])
• namespace:histogram(name[, help, buckets, metainfo])
• namespace:summary(name[, help, objectives, params, metainfo])

Collectors created through a namespace receive a selector built from the namespace selector and collector name:

<namespace-selector>.<collector-name>

Example:

local metrics = require('metrics')

local queue = metrics.namespace('app.queue')

local jobs_ready = queue:gauge(
    'jobs_ready',
    'Number of jobs ready for processing'
)

jobs_ready:set(5)

The collector selector is:

app.queue.jobs_ready

The metric can be enabled in the configuration:

metrics:
  include:
    - app.queue

Or more specifically:

metrics:
  include:
    - app.queue.jobs_ready

Add metrics.set_filter(include, exclude)

metrics.set_filter(include, exclude)

Set a runtime selector filter for custom metrics.

Parameters:

• include -- a table with selector objects to enable.
• exclude -- a table with selector objects to disable.

A selector object has the following format:

{selector = 'roles.crud-router'}

Example:

local metrics = require('metrics')

metrics.set_filter(
    {
        {selector = 'roles.crud-router'},
        {selector = 'app.queue'},
    },
    {
        {selector = 'roles.crud-router.crud_errors'},
    }
)

This runtime filter enables:

• custom metrics under roles.crud-router;
• custom metrics under app.queue.

It disables:

• roles.crud-router.crud_errors.

exclude has higher priority than include.

Declarative configuration is the recommended way to configure metrics in a Tarantool application. Use metrics.set_filter() when the selector filter has to be changed directly from Lua code.

Configuration reference updates

Update metrics.include

Suggested text:

metrics.include is an array containing built-in metric groups and custom metric selectors to turn on.

The array can contain the same values as the include configuration parameter passed to metrics.cfg():

• all
• none
• built-in metric group names, such as cpu, network, info, vinyl
• custom metric selectors, such as roles.crud-router or app.queue

A custom selector matches either exactly or by dot prefix.

Default: [ all ]

Environment variable: TT_METRICS_INCLUDE

Example:

metrics:
  include:
    - cpu
    - network
    - roles.crud-router
    - app.queue

Update metrics.exclude

Suggested text:

metrics.exclude is an array containing built-in metric groups and custom metric selectors to turn off.

The array can contain the same values as the exclude configuration parameter passed to metrics.cfg():

• built-in metric group names, such as cpu, network, info, vinyl
• custom metric selectors, such as roles.crud-router.crud_errors or app.queue.debug

A custom selector matches either exactly or by dot prefix.

metrics.exclude has higher priority than metrics.include.

Default: []

Environment variable: TT_METRICS_EXCLUDE

Example:

metrics:
  include:
    - all
    - roles.crud-router
  exclude:
    - vinyl
    - roles.crud-router.crud_errors

Start with the monitoring guide sections “Configuring metrics” and “Creating custom metrics”, then review the Lua metrics reference for metrics.cfg(), custom metrics, namespaces, and filters. Update the linked monitoring, Lua API, and configuration-reference sections with selector, namespace, callback, and include/exclude behavior examples; done means all listed documentation areas consistently describe the feature.