coder / coder/coder

feature: allow distinguishing between expected and unexpected provisioner disconnects

Open
#16,356 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.5k
Forks
1.5k
Avg merge
1d 20h
Merged PRs (30d)
601

Description

# Problem

Right now, we have no way of distinguishing between the following events:

- An external provisioner gets a normal signal to exit (for example, a user or deployment controller sends an interrupt signal)
- An external provisioner encounters a fatal error (panic, OOM) or otherwise exits unexpectedly

This makes it difficult for operators to troubleshoot issues with external provisioner daemons.

# Proposed Solution

Our schema currently looks something like this:

```
CREATE TYPE provisioner_daemon_status AS ENUM (
'offline',
'idle',
'busy'
);

CREATE TABLE provisioner_daemons (
id uuid NOT NULL,
created_at timestamp with time zone NOT NULL,
name character varying(64) NOT NULL,
provisioners provisioner_type[] NOT NULL,
replica_id uuid,
tags jsonb DEFAULT '{}'::jsonb NOT NULL,
last_seen_at timestamp with time zone,
version text DEFAULT ''::text NOT NULL,
api_version text DEFAULT '1.0'::text NOT NULL,
organization_id uuid NOT NULL,
key_id uuid NOT NULL
);
```

The logic for determining a provisioner daemon status is currently something like this:

```
CASE
WHEN pd.last_seen_at IS NULL OR pd.last_seen_at < (NOW() - (@stale_interval_ms::bigint || ' ms')::interval)
THEN 'offline'
ELSE CASE
WHEN current_job.id IS NOT NULL THEN 'busy'
ELSE 'idle'
END
END::provisioner_daemon_status AS status,
```

I initially started thinking about adding an extra field `exited_at` to the `provisioner_daemons` table with some logic around the exit status of the daemon, and modifying the logic.

@dannykopping made a very good observation though about how this is similar to the websocket status codes (e.g. NORMAL_CLOSURE, GOING_AWAY...). We could maybe include the websocket status code in the schema and use this as an indication of the exit reason.

cc @mafredri

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.