feature: allow distinguishing between expected and unexpected provisioner disconnects
- Dominant language
- Go
- Stars
- 14.5k
- Forks
- 1.5k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 601
Description
# Problem
Right now, we have no way of distinguishing between the following events:
- An external provisioner gets a normal signal to exit (for example, a user or deployment controller sends an interrupt signal)
- An external provisioner encounters a fatal error (panic, OOM) or otherwise exits unexpectedly
This makes it difficult for operators to troubleshoot issues with external provisioner daemons.
# Proposed Solution
Our schema currently looks something like this:
```
CREATE TYPE provisioner_daemon_status AS ENUM (
'offline',
'idle',
'busy'
);
CREATE TABLE provisioner_daemons (
id uuid NOT NULL,
created_at timestamp with time zone NOT NULL,
name character varying(64) NOT NULL,
provisioners provisioner_type[] NOT NULL,
replica_id uuid,
tags jsonb DEFAULT '{}'::jsonb NOT NULL,
last_seen_at timestamp with time zone,
version text DEFAULT ''::text NOT NULL,
api_version text DEFAULT '1.0'::text NOT NULL,
organization_id uuid NOT NULL,
key_id uuid NOT NULL
);
```
The logic for determining a provisioner daemon status is currently something like this:
```
CASE
WHEN pd.last_seen_at IS NULL OR pd.last_seen_at < (NOW() - (@stale_interval_ms::bigint || ' ms')::interval)
THEN 'offline'
ELSE CASE
WHEN current_job.id IS NOT NULL THEN 'busy'
ELSE 'idle'
END
END::provisioner_daemon_status AS status,
```
I initially started thinking about adding an extra field `exited_at` to the `provisioner_daemons` table with some logic around the exit status of the daemon, and modifying the logic.
@dannykopping made a very good observation though about how this is similar to the websocket status codes (e.g. NORMAL_CLOSURE, GOING_AWAY...). We could maybe include the websocket status code in the schema and use this as an indication of the exit reason.
cc @mafredri
Contributor guide
Assessment
This issue has not been assessed yet.