influxdata / influxdata/kapacitor

Simple TICKscript that will always error

Open
#1,217 21 comments 4 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Duplicate of https://github.com/influxdata/telegraf/issues/2444

>## Bug report
>
>Hey. For some reason, the system measurement (load1,5,15, uptime, etc) is sent by telegraf as two distinct lines.
>
>```
>system,host=TICKAlerta load1=0,load5=0.03,load15=0.03,n_users=1i,n_cpus=2i 1487579590000000000
>system,host=TICKAlerta uptime_format=" 0:13",uptime=807i 1487579590000000000
>```
>
>This essentially breaks stream processing with kapacitor of system measurement, as relevant field is missing 50% of the times.
>
>```
>E! error evaluating expression for level CRITICAL: no field or tag exists for load1
>```
>
>This can be verified while looking into datapoints, as seen by kapacitor
>
>```
>{
> "Name": "system",
> "Database": "telegraf",
> "RetentionPolicy": "default",
> "Group": "host=host1.ex",
> "Dimensions": {
> "ByName": false,
> "TagNames": [
> "host"
> ]
> },
> "Tags": {
> "environment": "offsite",
> "host": "host1.ex",
> "osname": "Ubuntu",
> "virtual": "physical"
> },
> "Fields": {
> "load1": 0,
> "load15": 0.05,
> "load5": 0.01,
> "n_cpus": 4,
> "n_users": 0
> },
> "Time": "2017-02-14T12:38:40Z"
>}
>{
> "Name": "system",
> "Database": "telegraf",
> "RetentionPolicy": "default",
> "Group": "host=host1.ex",
> "Dimensions": {
> "ByName": false,
> "TagNames": [
> "host"
> ]
> },
> "Tags": {
> "environment": "offsite",
> "host": "host1.ex",
> "osname": "Ubuntu",
> "virtual": "physical"
> },
> "Fields": {
> "uptime": 5278035,
> "uptime_format": "61 days, 2:07"
> },
> "Time": "2017-02-14T12:38:40Z"
>}
>```
>
>### Relevant telegraf.conf:
>
>```
>[global_tags]
>[agent]
> interval = "10s"
> round_interval = true
>
> metric_batch_size = 1000
>
> metric_buffer_limit = 10000
>
> collection_jitter = "0s"
>
> flush_interval = "10s"
> flush_jitter = "0s"
>
> precision = ""
>
> debug = false
> quiet = false
> logfile = ""
>
> hostname = ""
> omit_hostname = false
>
> [[outputs.file]]
> files = ["stdout", "/tmp/metrics.out"]
>
> data_format = "influx"
>
>[[inputs.system]]
> # no configuration
>```
>
>### System info:
>
>As far as I know, present in telegraf versions 1.0, 1.1 and 1.2. Tested on Ubuntu and Debian LTS versions (precise, trusty, xenial, jessie).
>
>### Steps to reproduce:
>
>Telegraf
>
>Use the included telegraf config file.
>```
>telegraf --config telegraf.conf --debug
>cat /tmp/metrics.out
>```
>
>Kapacitor
>```
>var warn_threshold = 4
>var crit_threshold = 10
>
>var period = 1h
>var every = 1m
>
>var data = stream
> |from()
> .database('telegraf')
> .retentionPolicy('default')
> .measurement('system')
> .groupBy('host')
> |log()
> |window()
> .period(period)
> .every(every)
> |last('load1')
> .as('stat')
>```
>```
>grep load1 /var/log/kapacitor/kapacitor.log
>```
>
>### Expected behavior:
>
>Single line for system measurement.
>
>### Actual behavior:
>
>Two distinct lines for system measurement
>

Since the data for a series is split into two lines, InfluxDB forwards it on to Kapacitor as two lines and therefore its possible to write a TICKscript that will perpetually fail

```js
var data = stream
|from()
.measurement('system')
|eval(lambda: "load1" / "uptime")
.as('ratio')
```
with
```
kapacitor define example -tick example.tick -dbrp telegraf.autogen -type stream
kapacitor enable example
```

will perpetually error out, even though all of the data would exist.

As I see it, there are four ways to solve this problem.

1. Kapacitor can do deduplication of incoming lines with common series key and timestamp.
2. InfluxDB can do deduplication of data it reports in subscriptions for lines with common series key and timestamp.
3. Telegraf can report data as a single line
4. We can provide documentation that notes this as a rough edge and give users a work around.

I had initially directed @markuskont to opening an issue on Telegraf, but there's definitely more than one way to solve this issue.

Contributor guide

Open the contributing guide

Research direction

Start with the reproduction entry points in telegraf.conf, the generated /tmp/metrics.out data, and example.tick with the shown Kapacitor define and enable commands. Compare the two system measurement lines and the failing load1/uptime evaluation; this issue is a duplicate and does not identify which of the four proposed systems should change.

Written by the indexing model from the issue text.

Assessment

Domain
stream-processing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.