0xfurai / 0xfurai/peekaping

Monitoring Tang (tangd) Endpoints creates lots of Zombie Processes - Does peekaping fail to close HTTP Connections ?

Ouverte
#272 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Go
Étoiles
1.2k
Forks
69
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

**Describe the bug**
Monitoring Tang (tangd) Endpoints creates lots of Zombie Processes.

I tried to configure some Overrides for `tangd.socket` using `systemctl edit tangd.socket`, but these fail to change the Behaviour and is therefore indicative that something is not working on the Tang Server too:
```
[Socket]
MaxConnections=256
RuntimeMaxSec=5m
```

Apparently `RuntimeMaxSec` is not a valid Setting, since I saw in the Journal that `/etc/systemd/system/tangd.socket.d/override.conf:5: Unknown key 'RuntimeMaxSec' in section [Socket], ignoring.`

(although that is **NOT** a valid Reason for `peekaping` to misbehave)

The Issue does **NOT** appear using the following alternative Monitors:
- Gatus
- Uptime Kuma
- CURL from my Desktop

Issue seems to exist both in 0.46.0 and 0.45.0, although in 0.45.0 it was "masked" since it would stay down forever (thus likely not triggering re-spawning of Zombie Processes on the Tang Server, see https://github.com/0xfurai/peekaping/issues/265).

Does `peekaping` fail to close HTTP Connections (properly) ?

**Database**
PostgreSQL

**To Reproduce**
Steps to reproduce the behavior:
1. Setup a Tang Server at
2. Monitoring Endpoint: `http://tang1.MYDOMAIN.TLD:80/adv`
3. Test using CURL: `curl -v --header "Content-Type: application/json" http://tang1.MYDOMAIN.TLD:80/adv`
4. Test using Peekaping using the Endpoint mentioned above and the following Settings:
Image

**Expected behavior**
Peekaping Requests should terminate correctly and the `tangd` spawn Process (on the Tang Server) shall terminate within a few Seconds.

**Screenshots**
N/A

**Desktop (please complete the following information):**
- OS: Ubuntu GNU/Linux 26.04 AMD64
- Browser: Librewolf 151.0-1 (based on Firefox 151.0)
- Version 26.04

**Smartphone (please complete the following information):**
N/A

**Additional context**
Test using CURL from Desktop
```
USER@ubuntu:~$ curl -v --header "Content-Type: application/json" http://tang1.MYDOMAIN.TLD:80/adv
* Host tang1.MYDOMAIN.TLD:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.15
* Trying 192.168.1.15:80...
* Established connection to tang1.MYDOMAIN.TLD (192.168.1.15 port 80) from 192.168.3.78 port 43178
* using HTTP/1.x
> GET /adv HTTP/1.1
> Host: tang1.MYDOMAIN.TLD
> User-Agent: curl/8.18.0
> Accept: */*
> Content-Type: application/json
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Type: application/jose+json
< Content-Length: 993
<
* Connection #0 to host tang1.MYDOMAIN.TLD:80 left intact
{"payload": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
```

`curl` on the Tang Server:
```
Jun 03 19:41:04 Tang1 systemd[1]: Started tangd@472-192.168.1.15:80-192.168.3.78:44066.service - Tang Server (192.168.3.78:44066).
░░ Subject: A start job for unit tangd@472-192.168.1.15:80-192.168.3.78:44066.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@472-192.168.1.15:80-192.168.3.78:44066.service has finished successfully.
░░
░░ The job identifier is 61018.
Jun 03 19:41:04 Tang1 tangd[4900]: 192.168.3.78 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:41:04 Tang1 systemd[1]: tangd@472-192.168.1.15:80-192.168.3.78:44066.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit tangd@472-192.168.1.15:80-192.168.3.78:44066.service has successfully entered the 'dead' state.

```

`peekaping` on the Tang Server:
```
Jun 03 19:42:00 Tang1 systemd[1]: Started tangd@475-192.168.1.15:80-192.168.1.68:49324.service - Tang Server (192.168.1.68:49324).
░░ Subject: A start job for unit tangd@475-192.168.1.15:80-192.168.1.68:49324.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@475-192.168.1.15:80-192.168.1.68:49324.service has finished successfully.
░░
░░ The job identifier is 61393.
Jun 03 19:42:00 Tang1 tangd[4917]: 192.168.1.68 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:43:00 Tang1 systemd[1]: Started tangd@476-192.168.1.15:80-192.168.1.68:35484.service - Tang Server (192.168.1.68:35484).
░░ Subject: A start job for unit tangd@476-192.168.1.15:80-192.168.1.68:35484.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@476-192.168.1.15:80-192.168.1.68:35484.service has finished successfully.
░░
░░ The job identifier is 61518.
Jun 03 19:43:00 Tang1 tangd[4920]: 192.168.1.68 GET /adv => 200 (../src/tangd.c:101)

```

(i.e. the spawn Process is NOT terminated)

`gatus` on the Tang Server:
```
Jun 03 19:41:58 Tang1 systemd[1]: Started tangd@473-192.168.1.15:80-192.168.1.136:33480.service - Tang Server (192.168.1.136:33480).
░░ Subject: A start job for unit tangd@473-192.168.1.15:80-192.168.1.136:33480.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@473-192.168.1.15:80-192.168.1.136:33480.service has finished successfully.
░░
░░ The job identifier is 61143.
Jun 03 19:41:58 Tang1 tangd[4909]: 192.168.1.136 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:41:58 Tang1 systemd[1]: tangd@473-192.168.1.15:80-192.168.1.136:33480.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit tangd@473-192.168.1.15:80-192.168.1.136:33480.service has successfully entered the 'dead' state.

```

`uptime-kuma` on the Tang Server:
```
Jun 03 19:42:00 Tang1 systemd[1]: Started tangd@474-192.168.1.15:80-192.168.1.59:41060.service - Tang Server (192.168.1.59:41060).
░░ Subject: A start job for unit tangd@474-192.168.1.15:80-192.168.1.59:41060.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tangd@474-192.168.1.15:80-192.168.1.59:41060.service has finished successfully.
░░
░░ The job identifier is 61268.
Jun 03 19:42:00 Tang1 tangd[4913]: 192.168.1.59 GET /adv => 200 (../src/tangd.c:101)
Jun 03 19:42:00 Tang1 systemd[1]: tangd@474-192.168.1.15:80-192.168.1.59:41060.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit tangd@474-192.168.1.15:80-192.168.1.59:41060.service has successfully entered the 'dead' state.
```

EDIT 1: Workaround to kill Zombie Tang Processes that are older than 5 Minutes

`/usr/local/bin/tang-kill-zombie-processes`:
```
#!/bin/bash
mapfile -t tang_processes < <( ps --no-headers -eo pid,etimes,uid,gid,cgroup,cgroupns,netns,pidns,userns,utsns,%cpu,%mem,pcpu,rss,vsz,cmd,comm | grep "/usr/libexec/tangd /var/lib tangd" | grep -v "grep" )

for tang_process in "${tang_processes[@]}"
do
pid=$(echo ${tang_process} | awk '{print $1}')
etimes=$(echo ${tang_process} | awk '{print $2}')

if [ ${etimes} -ge 600 ]
then
echo "Killing Process ${pid} which has been running for ${etimes}s"
kill -9 ${pid}
fi
done
```

`/etc/systemd/system/tang-kill-zombie-processes.service`:
```
[Unit]
Description=Kill Tang Zombie Processes

[Service]
ExecStart=/bin/bash -c '/usr/local/bin/tang-kill-zombie-processes'

[Install]
WantedBy=default.target
```

`/etc/systemd/system/tang-kill-zombie-processes.timer`:
```
[Unit]
Description=Kill Tang Zombie Processes
RefuseManualStart=no
RefuseManualStop=no

[Timer]
# Execute Job if it missed a run due to Machine being off
Persistent=true

# Set the timer accuracy to 15 Seconds instead of the default 1 Minute
AccuracySec=15s

# Run Every 5 Minutes
OnCalendar=*:0/5

# Run when enabled
OnActiveSec=5s

# File describing job to execute
Unit=tang-kill-zombie-processes.service

[Install]
WantedBy=timers.target
```

Make Script Executable:
```
chmod +x /usr/local/bin/tang-kill-zombie-processes
```

Reload Systemd Configuration
```
systemctl daemon-reload
```

Enable Timer:
```
systemctl enable tang-kill-zombie-processes.timer
```

Restart Timer:
```
systemctl restart tang-kill-zombie-processes.timer
```

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Look at the HTTP client code in peekaping, likely in a package handling probes or requests. Compare connection handling with other monitors like curl, gatus, and uptime-kuma. Reproduce the issue by setting up a local Tang server and monitoring it, then trace why connections are not closed. Check for proper use of http.Client, request cancellation, and response body closure.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
go, postgresql, shell
Domaine
backend, devops
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.