influxdata / influxdata/kapacitor
Kapacitor cpu usage on alerting
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi we have around 150 tasks that alerts every 10 minutes (test purpose)
the all looks like
```
var alert = data
|alert()
.id('{{index .TaskName }}')
.message('{{ .ID }} is {{ .Level }} value:{{ index .Fields "value" }} ')
.info(lambda: "value" < 1.0)
.crit(lambda: "value" >= 1.0)
// Alert
alert
.stateChangesOnly(600s)
.log('/var/log/stratoscale/strato-kapacitor.stratolog')
.exec('/kapacitor/kapacitor_queue_alert.py')
var dead = data
|deadman(0.0, 180s)
.id('{{index .TaskName }}__dead')
.message('{{index .TaskName }} is {{if eq .Level "OK"}}alive{{else}}dead{{end}}')
dead
.stateChangesOnly(600s)
.log('/var/log/stratoscale/strato-kapacitor.stratolog')
.exec('/kapacitor/kapacitor_queue_alert.py')
```
and we every 10 minutes when alert is spawned from all of them together we see cpu spikes that for 800% cpu usage.
Can we some configure kapacitor to limit numbers of concurrent tasks if it will help?
Or maybe "exec" is a problem?
Thanks a lot.
Contributor guide
Research direction
Start by reproducing the reported CPU spikes with approximately 150 alerting tasks using the shown alert, deadman, log, and exec configuration. Inspect the alerting and task-execution behavior when alerts are spawned together. Done means identifying whether concurrent tasks or exec handling causes the spikes and documenting a supported configuration or corrective change.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100