influxdata / influxdata/kapacitor

high CPU usage after SIGHUP reload

Open
#2,068 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Do you have ideas why sometimes after "reload" whole Kapacitor system blocked and CPU usage spike appears?

```
ts=2018-09-27T13:34:42.754Z lvl=info msg="SIGHUP received, reloading tasks/templates/handlers directory..." service=run
ts=2018-09-27T13:34:42.811Z lvl=info msg="stopped task" service=kapacitor task_master=main task=alert_cpu
ts=2018-09-27T13:34:42.851Z lvl=info msg="started task" service=kapacitor task_master=main task=alert_cpu
ts=2018-09-27T13:34:42.912Z lvl=info msg="stopped task" service=kapacitor task_master=main task=alert_disk
ts=2018-09-27T13:34:42.948Z lvl=info msg="started task" service=kapacitor task_master=main task=alert_disk
ts=2018-09-27T13:34:42.998Z lvl=info msg="stopped task" service=kapacitor task_master=main task=alert_disk_forecast
ts=2018-09-27T13:34:43.027Z lvl=info msg="started task" service=kapacitor task_master=main task=alert_disk_forecast
ts=2018-09-27T13:34:43.169Z lvl=info msg="stopped task" service=kapacitor task_master=main task=alert_influxdb_series
ts=2018-09-27T13:34:43.197Z lvl=info msg="started task" service=kapacitor task_master=main task=alert_influxdb_series
ts=2018-09-27T13:34:43.253Z lvl=info msg="stopped task" service=kapacitor task_master=main task=alert_iowait
ts=2018-09-27T13:34:43.289Z lvl=info msg="started task" service=kapacitor task_master=main task=alert_iowait
```

![adwaf214_](https://user-images.githubusercontent.com/6387460/46150680-bf39d280-c275-11e8-9532-0c72ed2ff08b.jpg)

seems whole system blocked for several minutes: it leads to False-positive deadman alerts trigger :(

note: if Reload kapacitor one more time after several minutes - everything is OK (cpu usage in normal state)

Contributor guide

Open the contributing guide

Research direction

The report only identifies SIGHUP reload behavior and includes Kapacitor logs; no source files or tests are named. Start by reproducing a reload while monitoring CPU and task activity, then trace the reload path and compare normal and high-CPU runs. Done means the cause is identified and reload no longer blocks the system or triggers false-positive deadman alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.