influxdata / influxdata/kapacitor

Question: Merging / aggregating grouped data

Open
#2,077 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

- I have several replica hosts that each report a metric value that is an event counter.
- Grouping by `host` allows me to calculate the derivative of this value for each host. This allows me to alert by host -- which is noisy and not really what I need.
- What I would like to be able to do is to apply an aggregate function across these hosts _after_ calculating the grouped (individual host) derivative.
- It seems there is no obvious way to do this _without_ using the `join` function. The `join` function would require I use several (~10) hardcoded/static queries, which is not ideal b/c the number of hosts may fluctuate and we don't want to constantly be editing the tick script to reflect this.

- Is there any other way?

```
// Omitting seemingly irrelevant vars from excerpted code
var groupList = ['name', 'host', 'region', 'dept']

var eventAcceptedCountData = task_type
|from()
.database(database)
.retentionPolicy(retention_policy)
.measurement(measurement)
.groupBy(groupList)
|default()
.field('EventAcceptedCount', 1.0)
|window()
.period(period)
.every(every)
|derivative('EventAcceptedCount')
.unit(1m)
.nonNegative()
.as('nnd')
|mean('nnd')
.as('eac_mean')
|eval(lambda: int("eac_mean"))
.as('eacmean')
|httpOut('eacmean')
```

I want to accomplish the same as above, but then follow up with a `sum` of `eacmean` _across_ hosts. If it isn't obvious, the derivative needs to be calculated by host because each host's event counter will differ due to age of the host instance and imperfect load-balancing.

Contributor guide

Open the contributing guide

Research direction

Start with the shown TICKscript pipeline, especially groupBy, derivative, join, and the final aggregation steps. Determine whether Kapacitor supports aggregating across host groups after per-host derivatives without static queries; done would be a confirmed supported approach or a clearly scoped change request.

Written by the indexing model from the issue text.

Assessment

Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.