Monitoring Tang (tangd) Endpoints creates lots of Zombie Processes - Does peekaping fail to close HTTP Connections ?
- Langage dominant
- Go
- Étoiles
- 1.2k
- Forks
- 69
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
**Describe the bug**
Monitoring Tang (tangd) Endpoints creates lots of Zombie Processes.
I tried to configure some Overrides for `tangd.socket` using `systemctl edit tangd.socket`, but these fail to change the Behaviour and is therefore indicative that something is not working on the Tang Server too:
```
[Socket]
MaxConnections=256
RuntimeMaxSec=5m
```
Apparently `RuntimeMaxSec` is not a valid Setting, since I saw in the Journal that `/etc/systemd/system/tangd.socket.d/override.conf:5: Unknown key 'RuntimeMaxSec' in section [Socket], ignoring.`
(although that is **NOT** a valid Reason for `peekaping` to misbehave)
The Issue does **NOT** appear using the following alternative Monitors:
- Gatus
- Uptime Kuma
- CURL from my Desktop
Issue seems to exist both in 0.46.0 and 0.45.0, although in 0.45.0 it was "masked" since it would stay down forever (thus likely not triggering re-spawning of Zombie Processes on the Tang Server, see https://github.com/0xfurai/peekaping/issues/265).
Does `peekaping` fail to close HTTP Connections (properly) ?
**Database**
PostgreSQL
**To Reproduce**
Steps to reproduce the behavior:
1. Setup a Tang Server at
2. Monitoring Endpoint: `http://tang1.MYDOMAIN.TLD:80/adv`
3. Test using CURL: `curl -v --header "Content-Type: application/json" http://tang1.MYDOMAIN.TLD:80/adv`
4. Test using Peekaping using the Endpoint mentioned above and the following Settings:
**Expected behavior**
Peekaping Requests should terminate correctly and the `tangd` spawn Process (on the Tang Server) shall terminate within a few Seconds.
**Screenshots**
N/A
**Desktop (please complete the following information):**
- OS: Ubuntu GNU/Linux 26.04 AMD64
- Browser: Librewolf 151.0-1 (based on Firefox 151.0)
- Version 26.04
**Smartphone (please complete the following information):**
N/A
**Additional context**
Test using CURL from Desktop
```
USER@ubuntu:~$ curl -v --header "Content-Type: application/json" http://tang1.MYDOMAIN.TLD:80/adv
* Host tang1.MYDOMAIN.TLD:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.15
* Trying 192.168.1.15:80...
* Established connection to tang1.MYDOMAIN.TLD (192.168.1.15 port 80) from 192.168.3.78 port 43178
* using HTTP/1.x
> GET /adv HTTP/1.1
> Host: tang1.MYDOMAIN.TLD
> User-Agent: curl/8.18.0
> Accept: */*
> Content-Type: application/json
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/jose+json
< Content-Length: 993
<
* Connection #0 to host tang1.MYDOMAIN.TLD:80 left intact
{"payload": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
```
`curl` on the Tang Server:
```
Jun 03 19:41:04 Tang1 systemd[1]: Started tangd@472-192.168.1.15:80-192.168.3.78:44066.service - Tang Server (192.168.3.78:44066).
░░ Subject: A start job for unit tangd@472-192.168.1.15:80-192.168.3.78:44066.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@472-192.168.1.15:80-192.168.3.78:44066.service has finished successfully.
░░
░░ The job identifier is 61018.
Jun 03 19:41:04 Tang1 tangd[4900]: 192.168.3.78 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:41:04 Tang1 systemd[1]: tangd@472-192.168.1.15:80-192.168.3.78:44066.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit tangd@472-192.168.1.15:80-192.168.3.78:44066.service has successfully entered the 'dead' state.
```
`peekaping` on the Tang Server:
```
Jun 03 19:42:00 Tang1 systemd[1]: Started tangd@475-192.168.1.15:80-192.168.1.68:49324.service - Tang Server (192.168.1.68:49324).
░░ Subject: A start job for unit tangd@475-192.168.1.15:80-192.168.1.68:49324.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@475-192.168.1.15:80-192.168.1.68:49324.service has finished successfully.
░░
░░ The job identifier is 61393.
Jun 03 19:42:00 Tang1 tangd[4917]: 192.168.1.68 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:43:00 Tang1 systemd[1]: Started tangd@476-192.168.1.15:80-192.168.1.68:35484.service - Tang Server (192.168.1.68:35484).
░░ Subject: A start job for unit tangd@476-192.168.1.15:80-192.168.1.68:35484.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@476-192.168.1.15:80-192.168.1.68:35484.service has finished successfully.
░░
░░ The job identifier is 61518.
Jun 03 19:43:00 Tang1 tangd[4920]: 192.168.1.68 GET /adv => 200 (../src/tangd.c:101)
```
(i.e. the spawn Process is NOT terminated)
`gatus` on the Tang Server:
```
Jun 03 19:41:58 Tang1 systemd[1]: Started tangd@473-192.168.1.15:80-192.168.1.136:33480.service - Tang Server (192.168.1.136:33480).
░░ Subject: A start job for unit tangd@473-192.168.1.15:80-192.168.1.136:33480.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@473-192.168.1.15:80-192.168.1.136:33480.service has finished successfully.
░░
░░ The job identifier is 61143.
Jun 03 19:41:58 Tang1 tangd[4909]: 192.168.1.136 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:41:58 Tang1 systemd[1]: tangd@473-192.168.1.15:80-192.168.1.136:33480.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit tangd@473-192.168.1.15:80-192.168.1.136:33480.service has successfully entered the 'dead' state.
```
`uptime-kuma` on the Tang Server:
```
Jun 03 19:42:00 Tang1 systemd[1]: Started tangd@474-192.168.1.15:80-192.168.1.59:41060.service - Tang Server (192.168.1.59:41060).
░░ Subject: A start job for unit tangd@474-192.168.1.15:80-192.168.1.59:41060.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@474-192.168.1.15:80-192.168.1.59:41060.service has finished successfully.
░░
░░ The job identifier is 61268.
Jun 03 19:42:00 Tang1 tangd[4913]: 192.168.1.59 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:42:00 Tang1 systemd[1]: tangd@474-192.168.1.15:80-192.168.1.59:41060.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit tangd@474-192.168.1.15:80-192.168.1.59:41060.service has successfully entered the 'dead' state.
```
EDIT 1: Workaround to kill Zombie Tang Processes that are older than 5 Minutes
`/usr/local/bin/tang-kill-zombie-processes`:
```
#!/bin/bash
mapfile -t tang_processes < <( ps --no-headers -eo pid,etimes,uid,gid,cgroup,cgroupns,netns,pidns,userns,utsns,%cpu,%mem,pcpu,rss,vsz,cmd,comm | grep "/usr/libexec/tangd /var/lib tangd" | grep -v "grep" )
for tang_process in "${tang_processes[@]}"
do
pid=$(echo ${tang_process} | awk '{print $1}')
etimes=$(echo ${tang_process} | awk '{print $2}')
if [ ${etimes} -ge 600 ]
then
echo "Killing Process ${pid} which has been running for ${etimes}s"
kill -9 ${pid}
fi
done
```
`/etc/systemd/system/tang-kill-zombie-processes.service`:
```
[Unit]
Description=Kill Tang Zombie Processes
[Service]
ExecStart=/bin/bash -c '/usr/local/bin/tang-kill-zombie-processes'
[Install]
WantedBy=default.target
```
`/etc/systemd/system/tang-kill-zombie-processes.timer`:
```
[Unit]
Description=Kill Tang Zombie Processes
RefuseManualStart=no
RefuseManualStop=no
[Timer]
# Execute Job if it missed a run due to Machine being off
Persistent=true
# Set the timer accuracy to 15 Seconds instead of the default 1 Minute
AccuracySec=15s
# Run Every 5 Minutes
OnCalendar=*:0/5
# Run when enabled
OnActiveSec=5s
# File describing job to execute
Unit=tang-kill-zombie-processes.service
[Install]
WantedBy=timers.target
```
Make Script Executable:
```
chmod +x /usr/local/bin/tang-kill-zombie-processes
```
Reload Systemd Configuration
```
systemctl daemon-reload
```
Enable Timer:
```
systemctl enable tang-kill-zombie-processes.timer
```
Restart Timer:
```
systemctl restart tang-kill-zombie-processes.timer
```
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Look at the HTTP client code in peekaping, likely in a package handling probes or requests. Compare connection handling with other monitors like curl, gatus, and uptime-kuma. Reproduce the issue by setting up a local Tang server and monitoring it, then trace why connections are not closed. Check for proper use of http.Client, request cancellation, and response body closure.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- go, postgresql, shell
- Domaine
- backend, devops
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100