PHP-FPM: Killing idle child issue using pm=ondemand

Hi

Introduction

Recently, I was playing around with PHP-FPM while using the process manager ondemand.

ondemand can only be used with kqueue (BSD) or epoll (Linux):

https://github.com/php/php-src/blob/d26068059e83fe40de3430a512471d194119bee0/sapi/fpm/fpm/fpm_conf.c#L922-L923

The following is a snippet of my pool configuration:

....
pm = ondemand
pm.max_children = 6
pm.process_idle_timeout = 30s;
pm.max_requests = 500

This means that each child is being killed if it has been idle for more than 30s. And this works as expected (when running 1 child).

# Last curl
Mon Nov 27 14:37:05 CET 2023

# Logs (30seconds after the last request)
[27-Nov-2023 14:37:36.344310] DEBUG: pid 26359, fpm_got_signal(), line 82: received SIGCHLD
[27-Nov-2023 14:37:36.351210] DEBUG: pid 26359, fpm_children_bury(), line 283: [pool xy-82] child 26360 has been killed by the process management after 99.362241 seconds from start

Problem

So this works like it's supposed to. The problem arises if I have a burst in requests, as seen in the screenshot:

In the beginning (after the burst), I have 3 running children:

1. PID: 38733 - 14:54:26 CET
2. PID: 39039 - 14:54:59 CET
3. PID: 39051 - 14:54:59 CET

After a minute, I still have 3 children. Even two processes (PID 39039, 39051) did not get any new requests (see counter)

After 4 minutes, I still have 2 children. Even PID 39051 did not get any new requests (see counter).

Reason

I dug into the php-fpm code and was able to find the problematic code snippet. On these lines, php-fpm tries to find the last_idle_child. For that, it iterates over his active children and if it is idle it tries to find the "oldest" idle child based on the ->started time. In my case, PID 38733 is idle and the oldest child and is therefore being selected as last_idle_child, even though it gets all the current requests (see counter).
https://github.com/php/php-src/blob/d26068059e83fe40de3430a512471d194119bee0/sapi/fpm/fpm/fpm_process_ctl.c#L361-L374

And there is the problem. Because of how epoll/kqueue works (Cloudflare Blog: Why does one NGINX worker take all the load?), it is possible that one child gets all the load/requests. The selected last_idle_child (PID 38733) is then checked to see if it exceeds the pm.process_idle_timeout. This is not the case for PID 38733, as it is the child which handles all the requests.

And that is the reason PID 39051 is not getting killed even it did not serve any request in the last 3 minutes.

https://github.com/php/php-src/blob/d26068059e83fe40de3430a512471d194119bee0/sapi/fpm/fpm/fpm_process_ctl.c#L388-L390

Expected behaviour

I would expect, that every child exceeding the pm.process_idle_timeout should be killed by the fpm-master. Regardless of whether it has been alive for a long time or not.

Possible Solution

It is not that easy to find a solution for this problem. I came up with this idea:
Check if pm.process_idle_timeout is reached for each child on every run (for ondemand) instead of picking the last_idle_child. This may be a bit more CPU intensive. Any other ideas?

PHP Version

PHP 8.2.13

Operating System

FreeBSD 12.4