Kapacitor Deadman switch false positives

Kapacitor fires fake alerts with deadman switch feature. We are running Kapacitor and InfluxDB in the same box.
We are pushing metrics from servers using telegraf every 60s. We have only 3 streams configured in Kapcitord;
* Usage Disk
* Cpu
* Deadman Switch

The first two work as expected, The third one is failing every so often.

If we stop telegraf agent we get a proper alert on time and when telegraf is back we get a second alert, but every so often we get alerts from other servers which did not fail. Sometimes is just one node, other times 20 or 30. Around 2 minutes later we get recovery alerts.

Kapacitor and telegraf running on debug so we can see the points are been pushed every minute but kapacitor keeps firing these alerts randomly.

**Operating system:** CentOS
**Full datails operating system:** Amazon Linux AMI release 2016.09 - 4.4.35-33.55.amzn1.x86_64
**Kapacitor Version:** Kapacitor 1.2.0 (git: master 5408057e5a3493d3b5bd38d5d535ea45b587f8ff) installed from .rpm
**Stream configuration**
```
ID: deadman-test-projectname
Error:
Template:
Type: stream
Status: enabled
Executing: true
Created: 22 Apr 17 09:48 UTC
Modified: 22 Apr 17 16:14 UTC
LastEnabled: 22 Apr 17 16:14 UTC
Databases Retention Policies: ["projectName"."only_one_month"]
TICKscript:
var id = '{{ index .Tags "host"}}'

var message_dead = '** PROBLEM Service Alert: {{ index .Tags "host" }} is DEAD'

var message_alive = '** Service Alert: {{ index .Tags "host" }} is RECEIVING METRICS AGAIN'

var data = stream
|from()
.database('projectName')
.retentionPolicy('only_one_month')
.groupBy(['environment', 'host'])
.where(lambda: TRUE)
|deadman(0.9, 120s, lambda: "environment" == 'test')
.id(id)
.message('{{ if eq .Level "OK" }}' + message_alive + '{{ else }}' + message_dead + '{{ end }}.')
.stateChangesOnly()
.hipChat()
.room('SaaS Alerts')
.pagerDuty()
.serviceKey('my_service_key')
|influxDBOut()
.database('kapacitor_alerts')

DOT:
digraph deadman-test-projectName {
graph [throughput="0.00 points/s"]; // Note: This value is different every time I run the command, its a coincidence is 0.00 now

stream0 [avg_exec_time_ns="0s" ];
stream0 -> from1 [processed="6177489"];

from1 [avg_exec_time_ns="35.168µs" ];
from1 -> noop3 [processed="6177489"];

noop3 [avg_exec_time_ns="0s" ];

stats2 [avg_exec_time_ns="199.096µs" ];
stats2 -> derivative4 [processed="111747"];

derivative4 [avg_exec_time_ns="2.048µs" ];
derivative4 -> alert5 [processed="111656"];

alert5 [alerts_triggered="697" avg_exec_time_ns="66.499µs" crits_triggered="348" infos_triggered="0" oks_triggered="349" warns_triggered="0" ];
alert5 -> influxdb_out6 [processed="697"];

influxdb_out6 [avg_exec_time_ns="16.541µs" points_written="697" write_errors="0" ];
}
```
Proof there are enough points to not fire an alert

A fresh example (sorry for the obfuscated names):
Log in kapacitor.log
Metrics received during two minutes with no issues and alert fired:
```
[httpd] 2017/04/24 09:07:00 D! write body received by handler: swap,dc=eu-west-1,environment=test,host=THIS_IS_THE_SERVER_NAME,influxdb_database_name=projectName free=0i,total=0i,used=0i,used_percent=0 1493024820000000000

2017/04/24 09:08:00 D! OK alert triggered id:THIS_IS_THE_SERVER_NAME msg:** Service Alert: THIS_IS_THE_SERVER_NAME is RECEIVING METRICS AGAIN. data:&{stats map[influxdb_database_name:projectName dc:eu-west-1 environment:test host:THIS_IS_THE_SERVER_NAME] [time emitted] [[2017-04-24 09:06:00 +0000 UTC 30]]}

[httpd] 2017/04/24 09:08:00 D! write body received by handler: swap,dc=eu-west-1,environment=test,host=THIS_IS_THE_SERVER_NAME,influxdb_database_name=projetName free=0i,total=0i,used=0i,used_percent=0 1493024880000000000
```

I hope you can shed some light on this.

Thanks,
Alejandro Gandara

Start with the supplied TICKscript, especially the stream groupBy and deadman(0.9, 120s) configuration, then inspect the Kapacitor debug log entries alongside the incoming Telegraf points. Reproduce the reported false-positive and recovery sequence, and consider the issue done when healthy hosts no longer alert while genuine Telegraf stoppages still produce timely alerts.

Written by the indexing model from the issue text.