Template task not triggering OK message, stops processing data completely when OK level is reached

Hello, I normally post in the community forums for support but I'm not sure it would be the right place. My use case with TICK is slightly different to most on there i think.

I'm trying to create a template that alerts to Nakivo backup job failures.

We have multiple instances of the TICK stack set up accross our clients. These instances uses Kapacitor/TICK script to forward data to a different Influx instance (hosted by us) - We do this so we can generate alerts and forward them to OpsGenie and a separate http endpoint. We do this because not all our clients allow access to OpsGenies URL (elastic IP's).

I have two instances of our TICK stack running locally, one of them is set to only forward data to my second instance. There is no alert logic involved on this instance. The TICK script is as simple as:
```

var db = 'SourceDB'
var rp = 'autogen'
var measurement string
// Output DB
var ccDatabase string
var ccRP = '24h'
var groupBy = ['cr_state','host','job_type','lr_date','lr_finish_date','lr_state','name','status','CustomerName']
var host string
var priority string

var whereFilter = lambda: ("job_type" == 'BACKUP' AND "host" == host)

var name = 'NakivoJobFailures'

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)

var prevensysWrite = data
|eval(lambda: "lr_vm_failed",
lambda: "vm_count",
)
.as('lr_vm_failed','vm_count')
.keep('lr_vm_failed','vm_count')
|influxDBOut()
.cluster('SecondDatabaseInstance')
.database(ccDatabase)
.retentionPolicy(ccRP)
.measurement(measurement)
```
When this template runs, data is sent to the second instance. I can see it arriving in the database. However if define my alert template on my second instance I have strange behaviour.

The template here streams the data as it is written into the database and applies alert logic to trigger an alert. If I trigger an alert, a critical alert is generated and written into my alerts database. As soon as i trigger an OK message the template stops processing data and no OK message is generated or written back to the alerts database. If i switch back to triggering a critical alert data beings processing again and i see new alerts being written into the database - I'm using a shell script with exec to insert test data for testing, but the result is the same using live data from our nakivo plugin.

The alert template:
```
var db = 'SourceDatabase'
var rp = '24h'
var measurement string
var groupBy = ['cr_state','host','job_type','lr_date','lr_finish_date','lr_state','name','status','CustomerName']
var host string
var priority string
var customer string
var whereFilter = lambda: ("CustomerName" == customer) AND ("host" == host)

var name = 'nakivotest'

var idVar = name + 'TEST'

var message = ' {{ index .Fields "value" }} {{ index .Tags "value" }}'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'Alerts'

var outputRP = 'autogen'

var outputMeasurement = 'NakivoAlerts'

var triggerType = 'threshold'

var crit = 0

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
|eval(lambda: "lr_vm_failed")
.as('value')

var trigger = data
|alert()
.crit(lambda: "lr_state" == 'FAILED' )
// .crit(lambda: "lr_vm_failed" >= 1)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
// .stateChangesOnly()
.log('/tmp/testnakivo.log')

trigger
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)

trigger
|httpOut('output')
```

Now, on Friday i was using a template i'd already created. Today i decided I'd go back to the start and create an alert using Chronograf. If i do this and start sending in the test data i get consistent alerts triggered, as soon as i set the test data to the "ok" data the Chronograf task processes the OK message and it is written back to the database.

The task from Chronograf was very basic, no extra nodes involed. As soon as i turn this into a temeplate (host, measurement and customer name become optional variables in the templates) I get the same behviour as before.

Critical alerts will trigger, but as soon as we send in our "ok" data the template stops processing.

Influx: 1.5.X
Kapacitor 1.5.2?

There are no Influx errors or Kapacitor errors that I can see in the log. No indication as to why the template stops processing data.

I've tried with the following thresholds

lr_vm_failed >= 1
lr_state == FAILED

These both trigger as normal, but i would expect an OK message when either lr_vm_failed was not greater than or equal to 1 OR when lr_state does not == FAILED

Our templates don't use a variable file to fill in the variables. We have a front end that queries the measurement for a key (host in this case) which lets us assign the host to the alert and we have an alerts page where we enter the thresholds required. This is controlled using JSON as meta data.

I don't know if using a separate variable file works, I'll test that next however a separate file for variables wouldn't work for us. We monitor hundreds of different systems, maintaining variable files for these across all clients would become messy and tedious and for our guys who are adding and monitoring these systems it makes their lives easier to manage and group alerts together, plus we want to avoid giving them multiple variable files, templates and access to the Kapacitor CLI.

Ayway, it just strikes me as odd that the template will generate the original alert message, it can obviously see the data as it arrives. Why doesn't it trigger the ok message?

I can post this on the forums if needed, but I'm not sure its within the scope of community support. Any ideas/suggestions?

Thanks,

PhilB

Edit: I was incorrect in my original post, data is still processed up to the alert node but never triggers the OK message.

Reproduce the issue by comparing the Chronograf-generated task with the templated task, using the alert node and template variables described in the report. Start with Kapacitor logs and alert state transitions while sending critical and OK data. Done means the templated task continues processing and writes the expected OK event to the alerts database.

Written by the indexing model from the issue text.