influxdata / influxdata/kapacitor

Kapacitor cpu usage on alerting

Open
#1,910 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi we have around 150 tasks that alerts every 10 minutes (test purpose)
the all looks like
```
var alert = data
|alert()
.id('{{index .TaskName }}')
.message('{{ .ID }} is {{ .Level }} value:{{ index .Fields "value" }} ')
.info(lambda: "value" < 1.0)
.crit(lambda: "value" >= 1.0)

// Alert
alert
.stateChangesOnly(600s)
.log('/var/log/stratoscale/strato-kapacitor.stratolog')
.exec('/kapacitor/kapacitor_queue_alert.py')

var dead = data
|deadman(0.0, 180s)
.id('{{index .TaskName }}__dead')
.message('{{index .TaskName }} is {{if eq .Level "OK"}}alive{{else}}dead{{end}}')

dead
.stateChangesOnly(600s)
.log('/var/log/stratoscale/strato-kapacitor.stratolog')
.exec('/kapacitor/kapacitor_queue_alert.py')

```
and we every 10 minutes when alert is spawned from all of them together we see cpu spikes that for 800% cpu usage.

Can we some configure kapacitor to limit numbers of concurrent tasks if it will help?
Or maybe "exec" is a problem?
Thanks a lot.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported CPU spikes with approximately 150 alerting tasks using the shown alert, deadman, log, and exec configuration. Inspect the alerting and task-execution behavior when alerts are spawned together. Done means identifying whether concurrent tasks or exec handling causes the spikes and documenting a supported configuration or corrective change.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.