influxdata / influxdata/kapacitor
Getting 'alert-name' is dead: %!f(MISSING) points/10s." data="&{stats map[] [time emitted] Error in Kapacitor
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We are trying to configure alert using Kapacitor.
Version: Kapacitor 1.4.0 (git: HEAD fcce3ee9e6abcee5595fd61066bfc904edb1e113)
Kapacitor alert script
```
var warn = 25
var crit = 50
var period = 60s
var every = 10s
// Dataframe
var data = batch
|query(''' SELECT used_percent AS stat FROM "infra-monitoring-production"."autogen"."disk" where ( host =~ /postgres/ OR host=~ /-db-/ OR host=~ /-pg-/) AND path=~ /\/var\/lib\/postgresql/ ''')
.period(period)
.every(every)
.groupBy('host')
// Thresholds
var alert = data
|alert()
.id('{{ index .Tags "host"}}/disk_total')
.message('{{ .ID }}:{{ index .Fields "stat" }}')
.warn(lambda: "stat" >= warn)
.crit(lambda: "stat" >= crit)
// Alert
alert
.post('http://some-webhook-url')
```
After defining and enabling the alert, we are getting below error from Kapacitor log.
```
ts=2019-01-08T19:59:10.007+07:00 lvl=debug msg="alert triggered" service=kapacitor task_master=main task=foobar1-alert node=alert5 level=CRITICAL id="node 'batch0' in task 'foobar1-alert'" event_message="node 'batch0' in task 'foobar1-alert' is dead: %!f(MISSING) points/10s." data="&{stats map[] [time emitted] [[2019-01-08 12:59:10 +0000 UTC 0]]}"
```
Although when we are checking data in Influxdb, data is present with correct measurement and field value.
```
> select * from "disk" where host='p-foobar-postgres-a-01' order by time DESC limit 10
name: disk
time device free fstype host inodes_free inodes_total inodes_used mode path total used used_percent
---- ------ ---- ------ ---- ----------- ------------ ----------- ---- ---- ----- ---- ------------
1546952590000000000 sdc 3415011328 ext4 p-foobar-postgres-a-01 655347 655360 13 /var/lib/postgresql 10434699264 6466039808 65.43878499365353
1546952590000000000 sda1 4035452928 ext4 p-foobar-postgres-a-01 1177494 1280000 102506 / 10340831232 6288601088 60.91212888129082
```
Strange things is that same Kapaitor script is working for other measurement and field value like CPU or mem, but not working for disk.
Any help on this on how to debug?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Kapacitor 1.4.0 task with the provided alert script and disk query, then compare the Kapacitor debug log with the InfluxDB disk results. Done means the task no longer reports the batch node as dead and the webhook receives the expected alert message.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100