Cog-Creators / Cog-Creators/Red-DiscordBot

Built-in liveness/readiness health endpoints (opt-in HTTP server)

Open
#6,802 2 comments 1 reaction 0 assignees View on GitHub
Category: Core - Other Internals Status: Needs Triage Type: Feature
Dominant language
Python
Stars
5.7k
Forks
2.5k
Avg merge
6d 16h
Merged PRs (30d)
1

Description

### Type of feature request

API functionality

### Description of the feature you're suggesting

Add an optional, built-in HTTP health server to Red core, enabled by a CLI flag
(mirroring how `--rpc` gates the RPC server), exposing standard **liveness** and
**readiness** endpoints for external monitoring - consumable by Kubernetes
liveness/readiness probes, load balancer health checks, and monitoring systems
(Prometheus, Datadog). These are the standard ways production services expose
whether they are alive and ready, and Red has no first-party way to do it today.

When started with a flag (e.g. `--health` / `--health-port`), Red would bind a
minimal HTTP listener and serve two endpoints, following the conventional
liveness/readiness split:

- `GET /health/live` - liveness: dependency-free, reflects only that the process
is alive and the event loop is responsive. Always cheap; never gated on Discord
connectivity (a transient gateway blip must **not** cause an orchestrator to
kill a healthy bot). Returns e.g. `200 {"status": "ok"}`.
- `GET /health/ready` - readiness: reports whether the bot is able to serve -
e.g. shard/connection state, `wait_until_red_ready()`, and the configured
Config datastore's reachability. Returns `200` when ready, `503` otherwise.

Both endpoints should exist from the first version even if `/health/ready`
initially mirrors the liveness response. The two carry different operational
semantics: a liveness failure tells an orchestrator to **restart** the process,
while a readiness failure tells it to **remove the instance from rotation** but
leave it running. Reserving both paths up front means operators wire their probes
at the correct endpoints immediately, and readiness can grow real dependency
checks later without ever changing the contract they already configured.

A natural later extension (out of scope for a first version) would let cogs
contribute named readiness checks that are aggregated into `/health/ready`.

The bind host/port should be configurable and default to loopback
(`127.0.0.1`), consistent with Red's existing security-conscious posture - the
RPC server is localhost-only "for security reasons". Operators running in a
container would opt into `0.0.0.0`.

This is new functionality, not an extension of something existing. Red's only
current external interface is the RPC server, which is websocket JSON-RPC bound
to `127.0.0.1:6133`, not configurable, and provisional - it cannot serve an HTTP
`GET` liveness/readiness probe from an orchestrator (Kubernetes kubelet), a load
balancer health check, a Docker healthcheck, or a monitoring system (Prometheus
blackbox exporter, Datadog HTTP check).

Liveness/readiness is deploy-time infrastructure, in the same category as the RPC
server - something an operator wires into *how the bot runs*, not a feature a
server admin toggles. That is why it belongs in core rather than a cog: a cog can
only approximate it (see below), and requiring a cog for basic operational health
is exactly the kind of capability self-hosted deployments (increasingly on
Kubernetes) expect from the platform itself.

### Anything else?

I've built this as a cog to prove the concept, and running it in production
surfaced precisely the limitations that argue for a core implementation:

- A cog has to bootstrap its own `aiohttp` server and manage the socket's
lifecycle by hand via `cog_load`/`cog_unload` - fragile plumbing to expect
every operator to install and babysit for something as fundamental as "is the
bot alive."
- A cog can't see the truest signals. Core knows shard/connection state,
`wait_until_red_ready()`, and the health of the configured Config datastore -
exactly the inputs a meaningful `/health/ready` should aggregate. A cog only
observes these indirectly.
- A cog-based endpoint only exists *after* the bot has connected and loaded cogs,
so it can't report health during early startup - which is exactly when an
orchestrator's startup probe wants a signal. A core server could answer
`/health/live` from process start.
- To be safe on a shared host the cog must default to `127.0.0.1`, then require a
manual reconfigure to `0.0.0.0` for containers. A flag-gated core feature could
handle this cleanly and document it once.

Reference implementation - a cog that serves liveness at `GET /health` via a
lifecycle-managed aiohttp server (started/stopped in `cog_load`/`cog_unload`,
loopback-default bind). It proves the liveness primitive in production; the
two-endpoint `/health/live` + `/health/ready` split described above is the shape
this proposal argues core should adopt (and where readiness gains access to the
signals a cog can't reach): **[shadow-templars/healthcheck](https://github.com/shadow-templars/shadow-cogs/tree/main/healthcheck)**

I'm happy to contribute a PR if the team is open to a flag-gated core health
server. I'd value guidance on the preferred shape: flag naming, endpoint paths
(`/health/live` + `/health/ready` vs. a single `/health`), what belongs in
readiness for a first version (shard state? Config datastore?), whether
cog-contributed readiness checks are in scope, and the default bind address.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Red core's existing --rpc startup path and the reference healthcheck cog, especially its cog_load/cog_unload lifecycle. Trace the available wait_until_red_ready and connection-state signals, then settle the flag, bind, endpoint, and readiness contract; done means an opt-in server exposes both documented HTTP endpoints with the specified liveness and readiness responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.