fpm: some workers get stuck on "\0fscf\0" write to master process over time

Description

After updating from PHP 7.3 to 8.1, we are now seeing the number of "active" php-fpm processes slowly grow over time, which is not expected. Looking at the fpm-status output, many processes report a duration of multiple days.

In the above, after restarting php-fpm, the number of active processes goes back down to around 60. So only about 30 got stuck out of 1 billion accepted connections. Really not that many, but resulting in +50% active processes after a week, and getting worse as time goes.

Looking at the stuck processes, all are on the exact same write. Here's one example:

# strace -p 5843
strace: Process 5843 attached
write(2, "\0fscf\0", 6

That fd is a pipe to the master php-fpm process:

php-fpm 5843 phpwww    2w  FIFO               0,13       0t0 2303333681 pipe

php-fpm    5843          phpwww    2w     FIFO               0,13       0t0 2303333681 pipe
php-fpm   91227            root 1392r     FIFO               0,13       0t0 2303333681 pipe

91227 ?        Ss   339:39 php-fpm: master process (/etc/php-fpm1.conf)
79811 ?        S      1:37  \_ php-fpm: pool www1
80146 ?        S      1:38  \_ php-fpm: pool www1
80914 ?        S      1:36  \_ php-fpm: pool www1
[...]

The fscf string can be found in sapi/fpm/fpm/fpm_stdio.c:

#define FPM_STDIO_CMD_FLUSH "\0fscf"

So for some reason, it looks like in some rare cases, sending this command from a worker back to the master gets stuck forever.

PHP Version

PHP 8.1.20

Operating System

RHEL 7.9