squid: FATAL: assertion failed: mem/PageStack.cc:159: "StoredNode().is_lock_free()" when workers enabled

Maintainer: @krant
Environment: x86 (APU, SuperMicro, Generic), ramips (unielec u7621-06, Mikrotik RB750Gr3), 23.05.x and snapshot

Squid supports workers to run parallel kids on multiple cores to utilise SMP. It was working fine until squid version 4.x but stopped working with squid 5.x and squid 6.x on Openwrt.

To enable workers following changes are required:

/etc/squid/squid.conf:

workers 2
# optionally bind processes to specific cores:
cpu_affinity_map process_numbers=1,2 cores=1,3 

/etc/init.d/squid:

## replace -N with --foreground
procd_set_param command $PROG -s -f $CONFIGFILE --foreground 

With multiple kids, processes are seen as below:

# ps ax | grep squid
2997 pts/0    S+     0:00 grep squid
 8815 ?        S      0:04 /usr/sbin/squid -s -f /tmp/squid/squid.conf --foreground
 9132 ?        S      1:55 (squid-coord-3) --kid squid-coord-3 -s -f /tmp/squid/squid.conf --foreground
 9133 ?        S    114:19 (squid-2) --kid squid-2 -s -f /tmp/squid/squid.conf --foreground
 9134 ?        S    113:01 (squid-1) --kid squid-1 -s -f /tmp/squid/squid.conf --foreground

But with newer squid (5.x and 6.x) on Openwrt 23.05.0 onwards fail to start when workers are enabled (I haven't tested on 22.03.x as jumped to 23.05.x from 21.02.x):

Thu Jun 27 07:07:11 2024 daemon.notice squid[3357]: Processing Configuration File: /tmp/squid/squid.conf (depth 0)
Thu Jun 27 07:07:11 2024 daemon.notice squid[3357]: ERROR: 'pinger_enable' requires --enable-icmp
Thu Jun 27 07:07:12 2024 local4.notice squid[3357]: Created PID file (/var/run/squid.pid)
Thu Jun 27 07:07:12 2024 local4.warn squid[3357]: FATAL: assertion failed: mem/PageStack.cc:159: "StoredNode().is_lock_free()"

Below is the squid debug log with debug_options 54,9:

2024/06/27 07:10:41| Processing Configuration File: /tmp/squid/squid.conf (depth 0)
2024/06/27 07:10:41| ERROR: 'pinger_enable' requires --enable-icmp
2024/06/27 07:10:42| Created PID file (/var/run/squid.pid)
2024/06/27 07:10:42.385| 54,3| mem/Segment.cc(245) unlink: unlinked /squid-cf__metadata.shm segment
2024/06/27 07:10:42.385| 54,3| mem/Segment.cc(128) create: created /squid-cf__metadata.shm segment: 8
2024/06/27 07:10:42.385| 54,5| mem/Segment.cc(211) lock: mlock(2)-ing disabled
2024/06/27 07:10:42.386| 54,3| mem/Segment.cc(245) unlink: unlinked /squid-cf__queues.shm segment
2024/06/27 07:10:42.386| 54,3| mem/Segment.cc(128) create: created /squid-cf__queues.shm segment: 73912
2024/06/27 07:10:42.386| 54,5| mem/Segment.cc(211) lock: mlock(2)-ing disabled
2024/06/27 07:10:42.387| 54,3| mem/Segment.cc(245) unlink: unlinked /squid-cf__readers.shm segment
2024/06/27 07:10:42.387| 54,3| mem/Segment.cc(128) create: created /squid-cf__readers.shm segment: 56
2024/06/27 07:10:42.387| 54,5| mem/Segment.cc(211) lock: mlock(2)-ing disabled
2024/06/27 07:10:42.387| 54,7| Queue.cc(50) QueueReader: constructed ipcQR1
2024/06/27 07:10:42.388| 54,7| Queue.cc(50) QueueReader: constructed ipcQR2
2024/06/27 07:10:42.388| 54,7| Queue.cc(50) QueueReader: constructed ipcQR3
2024/06/27 07:10:42.388| 54,5| mem/PageStack.cc(129) IdSetMeasurements: rounded capacity up from 8192 to 8192
2024/06/27 07:10:42.388| 54,3| mem/Segment.cc(245) unlink: unlinked /squid-squid-page-pool.shm segment
2024/06/27 07:10:42.389| 54,3| mem/Segment.cc(128) create: created /squid-squid-page-pool.shm segment: 268437592
2024/06/27 07:10:42.389| 54,5| mem/Segment.cc(211) lock: mlock(2)-ing disabled
2024/06/27 07:10:42.389| 54,5| mem/PageStack.cc(129) IdSetMeasurements: rounded capacity up from 8192 to 8192
2024/06/27 07:10:42| FATAL: assertion failed: mem/PageStack.cc:159: "StoredNode().is_lock_free()"

Reproduce on OpenWrt with workers 2 and the shown /etc/init.d/squid --foreground change, collecting debug_options 54,9 output. Then inspect mem/PageStack.cc around line 159 and the reported shared-memory setup; done means Squid starts with workers enabled without the lock-free assertion.