0xfurai / 0xfurai/peekaping

Peekaping is stuck once a monitor goes down (for whatever Reason)

オープン
#265 コメント 5 件 リアクション 1 件 担当者 0 名 GitHub で見る
主要言語
Go
スター
1.2k
フォーク
69
PR マージ指標
30日以内にマージされた PR はありません

説明

**Describe the bug**
Peekaping is stuck once a monitor goes down (for whatever Reason). I'm not entirely sure why, but once a monitor (Ping or HTTP) goes down, it doesn't come back up (almost ever).

I have also `gatus` and `uptime-kuma` monitoring those Hosts so I know for a Fact that most of them are 100% up.

Regardless of why the monitor came down, it should continuously be pinged (ICMP Ping) / queried (HTTP). Instead, after a while, **Peekaping just gives up even trying anymore**, thus the Monitor is and stays down forever.

**Database**
- postgresql

**Version**
- Peekaping: 0.0.45

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
Monitor should continuously be pinged (ICMP Ping) / queried (HTTP) even after it goes down.

This is what `gatus` and `uptime-kuma` do out of the Box.

Peekaping instead seems to get stuck and/or give up after a while 😕.

**Screenshots**
Time is currently between 08h06 (when I started writing this Issue) and 08h22 (when I finished writing this Issue).

Ping Monitor: Peekaping has been stuck / locked for the past 1+ Hour:

Image

```
podman@peekaping:~/containers/quadlets/peekaping$ journalctl --user -xeu peekaping-ingester --no-pager | grep -i "jk-bms-bat01"
Mar 08 06:09:22 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946562.0153909,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.011741054}
Mar 08 06:09:41 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946581.5432782,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.003792322}
Mar 08 06:10:01 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946601.3335428,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.029473007}
Mar 08 06:10:21 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946621.4557583,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.019578605}
Mar 08 06:10:40 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946640.2602067,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.013078612}
Mar 08 06:11:01 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946661.2837162,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.02184739}
Mar 08 06:11:21 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946681.1704283,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.013652279}
Mar 08 06:11:40 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946700.7179687,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.019816899}
Mar 08 06:12:01 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946721.4598243,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.024424851}
Mar 08 06:12:21 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946741.967585,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.006606443}
Mar 08 06:12:40 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946760.8822973,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.015436036}
Mar 08 06:13:00 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946780.4174168,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.017082199}
Mar 08 06:13:20 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946800.9011767,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.013009503}
Mar 08 06:13:41 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946821.9405189,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.020048074}
Mar 08 06:14:00 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946840.8907442,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.018757894}
Mar 08 06:14:21 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946861.033639,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.012397571}
Mar 08 06:14:41 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946881.866438,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.013919811}
Mar 08 06:15:01 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946901.4896088,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.002308295}
Mar 08 06:15:21 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946921.9233463,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.012852423}
Mar 08 06:15:41 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946941.1913452,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.038515999}
Mar 08 06:16:01 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946961.8891952,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.00965311}
Mar 08 06:16:20 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772946980.8768117,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.018421893}
Mar 08 06:16:41 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772947001.5268166,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.006403313}
Mar 08 06:17:02 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772947022.151422,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.027069306}
Mar 08 06:17:22 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772947042.1583946,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.016098031}
Mar 08 06:17:41 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772947061.6278179,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.004193929}
Mar 08 06:18:01 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772947081.3016024,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.019444969}
Mar 08 06:18:38 peekaping peekaping-ingester[125362]: {"level":"info","ts":1772947118.0940034,"caller":"ingester/handler.go:96","msg":"Successfully processed ingester task","svc":"peekaping:ingester","component":"ingester_handler","monitor_id":"139b03a6-1a34-42e8-86b4-74a989f1b6e9","monitor_name":"jk-bms-bat01","duration":0.011185254}

podman@peekaping:~/containers/quadlets/peekaping$ date
Sun Mar 8 08:15:57 AM CET 2026
```

Image

HTTP Monitor:Peekaping has been stuck / locked for the past 7+ Hour:

Image

Image

For some weird Reason it says the Certificate Expiry Notification is enabled even though it clearly is not:
```
podman@peekaping:~/containers/quadlets/peekaping$ journalctl --user -xeu peekaping-producer --no-pager | grep -i "tang1"
Mar 08 08:07:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953620.0261664,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:07:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953650.026036,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:08:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953680.0305603,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:08:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953710.0372384,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:09:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953740.010066,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:09:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953770.0203052,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:10:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953800.0368657,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:10:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953830.043507,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:11:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953860.054222,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:11:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953890.0141444,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:12:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953920.0385141,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:12:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953950.041805,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:13:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772953980.0254714,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:13:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954010.0441544,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:14:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954040.052076,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:14:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954070.0187242,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:15:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954100.0201797,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:15:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954130.014118,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:16:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954160.0591426,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:16:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954190.0177045,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:17:00 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954220.0287147,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}
Mar 08 08:17:30 peekaping peekaping-producer[125344]: {"level":"debug","ts":1772954250.0288277,"caller":"producer/processor.go:197","msg":"Certificate expiry checking configured","svc":"peekaping:producer","component":"producer","monitor_id":"f36d7225-4e81-4a0f-8d66-f154b347c477","monitor_name":"tang1","check_cert_expiry":false}

podman@peekaping:~/containers/quadlets/peekaping$ date
Sun Mar 8 08:17:54 AM CET 2026
```

Image

**Desktop (please complete the following information):**
- OS: Ubuntu GNU/Linux 24.04 AMD64
- Browser: Brave 1.87.192 (Official version) (64-bit), based on Chromium 145.0.7632.160

**Smartphone (please complete the following information):**
N/A (only using Desktop for now)

**Additional context**
Add any other context about the problem here.

EDIT 1: from inside the Containers / Pod, after installing `curl`, I can `curl` the HTTP Endpoints without Issue. `ping` works fine. DNS Resolution works without Issues.

So ... it's something wrong with Peekaping's Binary 😕.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

The issue points to the ingester component (ingester/handler.go) and producer (producer/processor.go). Start by examining the task scheduling logic in the ingester to see why it stops processing after a monitor goes down. Look for any state that might cause tasks to be dropped or paused. Check the database for monitor status flags. Reproduce by setting up a test monitor and simulating a failure.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
go, postgresql
領域
backend, observability
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。