influxdata / influxdata/kapacitor
Question: Merging / aggregating grouped data
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
- I have several replica hosts that each report a metric value that is an event counter.
- Grouping by `host` allows me to calculate the derivative of this value for each host. This allows me to alert by host -- which is noisy and not really what I need.
- What I would like to be able to do is to apply an aggregate function across these hosts _after_ calculating the grouped (individual host) derivative.
- It seems there is no obvious way to do this _without_ using the `join` function. The `join` function would require I use several (~10) hardcoded/static queries, which is not ideal b/c the number of hosts may fluctuate and we don't want to constantly be editing the tick script to reflect this.
- Is there any other way?
```
// Omitting seemingly irrelevant vars from excerpted code
var groupList = ['name', 'host', 'region', 'dept']
var eventAcceptedCountData = task_type
|from()
.database(database)
.retentionPolicy(retention_policy)
.measurement(measurement)
.groupBy(groupList)
|default()
.field('EventAcceptedCount', 1.0)
|window()
.period(period)
.every(every)
|derivative('EventAcceptedCount')
.unit(1m)
.nonNegative()
.as('nnd')
|mean('nnd')
.as('eac_mean')
|eval(lambda: int("eac_mean"))
.as('eacmean')
|httpOut('eacmean')
```
I want to accomplish the same as above, but then follow up with a `sum` of `eacmean` _across_ hosts. If it isn't obvious, the derivative needs to be calculated by host because each host's event counter will differ due to age of the host instance and imperfect load-balancing.
Contributor guide
Research direction
Start with the shown TICKscript pipeline, especially groupBy, derivative, join, and the final aggregation steps. Determine whether Kapacitor supports aggregating across host groups after per-host derivatives without static queries; done would be a confirmed supported approach or a clearly scoped change request.
Written by the indexing model from the issue text.
Assessment
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100