Need alternative HealthCheck

What happened?

Thanks to my recent task to check that my MariaDB backups are working fine (had issues with Ofelia not firing, hence I switched to Supercronic), I noticed issues on my server to day. I saw an hour old error.flag file, which normally creates an email, which is sent by a job, that's triggered by Supercronic on FrankenPHP container, if that container is healthy. But it was not triggered, because the container was not healthy. Although it seemed to be working fine.

Current healthcheck calls curl -f http://localhost:2019/metrics, and I did that. I got a flood of entries like

caddy_rate_limit_process_time_seconds_sum{key="130.180.236.170/32",zone="reads"} 5.0729e-05
caddy_rate_limit_process_time_seconds_count{key="130.180.236.170/32",zone="reads"} 1
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.001"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.005"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.01"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.025"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.05"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.1"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.25"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="0.5"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="1"} 2
caddy_rate_limit_process_time_seconds_bucket{key="130.180.236.176/32",zone="reads",le="+Inf"} 2
caddy_rate_limit_process_time_seconds_sum{key="130.180.236.176/32",zone="reads"} 7.129200000000001e-05

It continued for like 20 seconds, when I broke it. I did a docker inspect --format='{{json .State.Health}}' frankenphp and it last healthcheck exceeded 30s timeout. But that's not all: looks like this also resulted in OOM that killed mariadb container. I do not have hard proof of that, but OOM did kill mariadb, and the timing is mighty suspicious. I see there was also a https://github.com/php/frankenphp/issues/439, which was caused by a memory leak, but it does suggest that healthcheck can cause a spike in memory usage, and if it will trigger OOM-killer, mariadb will be selected since it uses the most of RAM.

The entries are likely caused by https://github.com/mholt/caddy-ratelimit so I disabled metrics there for now (at least), but... Someone may need these metrics. Maybe not from this plugin, but from something else, that can also result in growth of the output from /metrics endpoint. Maybe not to the same extent, as rate-limiter, but depending on what else is happening on the server even a small increase in size can trigger an OOM or slowness of output of the metrics and you get issues.

As such, I am raising this ticket so that an alternative healthcheck is considered in the FrankenPHP Docker image. And since this is built into FrankenPHP image, I am raising it for FrankenPHP. It also may be worthwhile to have something that not only checks that Caddy is up, but that PHP interpreter is working, too. For Caddy side it looks like /config endpoint is the most neutral, but my config is 122KBs, which is relatively large, too, so may not be ideal.

Build Type

Docker (Debian Trixie)

Worker Mode

No

Operating System

GNU/Linux

CPU Architecture

x86_64

PHP configuration

N/A

Relevant log output

N/A

Start by locating the FrankenPHP Docker image definition and its current healthcheck, which calls the Caddy /metrics endpoint. Compare possible checks against the reported /config and PHP-interpreter requirements. Done should mean a healthcheck that avoids unbounded metrics output while reliably detecting whether Caddy and PHP are working.