influxdata / influxdata/kapacitor
DEADMan scrip for no data
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Desperate help needed soon!
I am trying to use DEADMAN with no avail.
I am trying to convince the large corporation i work at, with thousands of servers to user Telegraph for collection Into and Influx database and Chronograf to write tick scripts for Alerting
I am simply trying to write a DEADMAN script that report on Data no longer being collected, that reports the HOST/Database name when data is no longer being collected. And when the collection stop during an interval. It fires OK, but does not give me the tags.
Plus I am trying to report on No data collected for servers that have no data in the past couple of weeks. prior to my writing of the script. These may have been decommissioned servers, or collectD was not started on the server to send data to Influx, etc. If decommissioned, we will clean out the Influx Database and forget about the data. For collectD not running, i want to alert the people responsible to fix this issue. Other alerts will be dealt with if the 2 cases above are not the case
With all the confusion and having to write two streams etc, and the group by issue that i have read in every post i have found, couldn't you just allow a collection on point, lets say say cpu_percent , or another metrics, and trigger a no data is returned and has 0 points from the measurement. not the value 0 but 0 points or a no data returned at all flag.
I am having the same issues trying to use the Tick Script for deadman on 1000+ servers monitoring CPU. It does not seem to trigger the event for NO DATA RETURNED.
I have tried group by on the Tags, Group by *, no group by and the deadman does not seem to trigger.
I really don't have to write a new rule for 1000+ servers or databases. and especially don't want to write a new rule each time a server or database is added to the collection. I just want to group by the server name and not have to go into chronograph rule and select any new additional servers. That would automate the addition of any new servers / databases
for the no data situation, If I could just use a stream collector and trigger my own e-mail / stack etc, saying No metric data collected in the past xx minutes and go from there.
Either Deadman need fixed, or further explained on how to group by or not group by, so the event triggers on no data as well as data now coming in, or we need a NO DATA RETURNED flag in Chronograf / Kapacitor to Simulate what we are trying to accomplish with DEADMAN.
I really don't want to have a bunch of people learn native Kapacitor, and would like to allow people to develop their own altering rules using Chronograf ONLY!
Contributor guide
Research direction
Start by reproducing the reported Deadman behavior for CPU data across grouped hosts and databases, including servers with no recent points. Review the Kapacitor and Chronograf Deadman and no-data behavior described in the issue. Done means the intended no-data condition is either supported and documented for these cases or the requested behavior is clearly scoped for implementation.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100