Health checks: liveness and readiness probes

Related dev. issue(s): https://github.com/tarantool/tarantool/pull/12681
Product: Tarantool
Since: 3.8.0
Root document:

• box.info reference:
  https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_info/info/
• box.ctl reference:
  https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_ctl/
• Application roles:
  https://www.tarantool.io/en/doc/latest/platform/app/app_roles/
• box.info.config reference, for health alerts integration:
  https://www.tarantool.io/en/doc/latest/reference/reference_lua/box_info/config/

SME: @mandesero

Details

• Document the new box.info.health field. It contains two sections:
  liveness and readiness.

• Document liveness checks:

  • box.info.health.liveness.verdict is true only if all registered
    liveness checks are successful.
  • box.info.health.liveness.checks is a map of check names to check results.
  • Each liveness check has status = 'ok' or status = 'failed'.
  • Failed checks may contain reason and alert_code.
  • There is no top-level reason in liveness; reasons belong to individual
    checks.

• Document box.ctl.liveness_probe(opts):

  • opts.name is the check name.
  • opts.check is a callback returning true on success or
    false/nil, <reason> on failure.
  • opts.alert = false disables config alerts for this liveness check.
  • opts.alert_code overrides the default alert code.
  • The default alert code is health.liveness.<name>.

• Document readiness checks:

  • box.info.health.readiness.status is true only if all readiness checks
    are ready.
  • box.info.health.readiness.checks is a map of check names to check
    results.
  • Each readiness check has status = 'ready', status = 'not_ready', or
    status = 'degraded'.
  • Initially, a failing readiness check is not_ready.
  • If a check was ready and then starts failing, its status becomes
    degraded.

• Document built-in readiness checks:

  • box: ready when box.info.status == 'running'.
  • config: not ready on startup_in_progress and check_errors; config
    warnings do not make the instance not ready.
  • vshard.router: ready when router bucket discovery is complete.

• Document the application role contract extension:

  • A role may define health_check(cfg).
  • The method is called as part of readiness evaluation.
  • It receives the role configuration from roles_cfg.
  • It should return true if the role is ready, or false/nil, <reason> if
    it is not ready.
  • The corresponding readiness check name is role.<role_name>.

• Document error handling:

  • Health check callbacks are executed safely.
  • If a callback raises an error, the corresponding check is reported as
    failed/not ready/degraded with the error as the reason.
  • Health check callbacks must not yield. Yielding checks are reported as
    failed/not ready/degraded with the reason health check must not yield.
  • Recursive health check evaluation is forbidden.

• Document config alerts integration:

  • Failed liveness and readiness checks are reflected in
    box.info.config.alerts.
  • Alert messages identify the failed check and include the failure reason.
  • Alerts are removed when the check becomes successful or is unregistered.

Examples

Liveness probe:

box.ctl.liveness_probe({
    name = 'event-loop',
    check = function()
        return true
    end,
})

Liveness probe with a custom alert code:

box.ctl.liveness_probe({
    name = 'watchdog',
    check = function()
        return false, 'watchdog stopped'
    end,
    alert_code = 'app.watchdog.stopped',
})

Liveness probe without config alerts:

box.ctl.liveness_probe({
    name = 'diagnostic',
    check = function()
        return false, 'diagnostic failure'
    end,
    alert = false,
})

Role readiness check:

return {
    validate = function(cfg)
        -- Validate role configuration.
    end,

    apply = function(cfg)
        -- Apply role configuration.
    end,

    stop = function()
        -- Stop the role.
    end,

    health_check = function(cfg)
        if not cfg.enabled then
            return false, 'role is disabled'
        end
        return true
    end,
}

Health output example:

box.info.health
-- {
--     liveness = {
--         verdict = true,
--         checks = {
--             ['event-loop'] = {
--                 status = 'ok',
--             },
--         },
--     },
--     readiness = {
--         status = false,
--         checks = {
--             box = {
--                 status = 'ready',
--             },
--             ['role.my_role'] = {
--                 status = 'not_ready',
--                 reason = 'role is disabled',
--                 alert_code = 'health.readiness.role.my_role',
--             },
--         },
--     },
-- }

Start with the linked box.info, box.ctl, application roles, and box.info.config reference pages, then locate the corresponding documentation sections for health, probes, and role configuration. Document the liveness and readiness fields, built-in checks, role health_check contract, error handling, alerts, and the supplied Lua examples; done means each listed behavior is covered consistently in the reference documentation.