influxdata / influxdata/kapacitor

Aggregating data with multiple fields

Open
#696 5 comments 3 reactions 0 assignees View on GitHub
new-feature
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I'm trying to aggregate all fields of a certain series to time steps of 1h. With continuous queries, this worked as follows:

```
CREATE CONTINUOUS QUERY cq_1h_net_sec ON telegraf
BEGIN
SELECT
mean(bytes_sec_recv) as bytes_sec_recv_avg,
max(bytes_sec_recv) as bytes_sec_recv_max,
min(bytes_sec_recv) as bytes_sec_recv_min,
percentile(bytes_sec_recv, 95) as bytes_sec_recv_perc95,
mean(bytes_sec_sent) as bytes_sec_sent_avg,
max(bytes_sec_sent) as bytes_sec_sent_max,
min(bytes_sec_sent) as bytes_sec_sent_min
percentile(bytes_sec_sent, 95) as bytes_sec_sent_perc95,
INTO "store1y"."net_sec"
FROM "store7d".net_sec
GROUP BY *, time(1h)
END
```

In the example, I process two fields, but I'm searching for a more general solution that processes all fields, a regex match or the like. With kapacitor, I don't see a solution yet; maybe splitting, processing and joining fields?

```
var val_raw = stream
|from()
.database('telegraf')
.retentionPolicy('store7d')
.measurement('net_sec')
.groupBy(*)
|default()
.field('bytes_sec_recv', 0)
.field('bytes_sec_sent', 0)

var windowed_vals = val_raw
|window()
.period(30s)
.every(30s)
.align()

var bytes_sec_recv_avg = windowed_vals
|mean('bytes_sec_recv')
.as('bytes_sec_recv_avg')

var bytes_sec_sent_avg = windowed_vals
|mean('bytes_sec_sent')
.as('bytes_sec_sent_avg')

bytes_sec_recv_avg
|union(bytes_sec_sent_avg)
|influxDBOut()
.database('telegraf')
.retentionPolicy('store1y')
.measurement('net_sec')
.precision('s')
```

Is this the way to go? I'm not sure if this scales well, as there are ~8 fields (rx and tx of bytes, packets, errors and drops) and 6 aggregation functions (min, max, avg, percentile (25, 75, 95)) per telegraf net series, leading to 48 variables...
Something like

```
|mean('bytes_sec_sent', 'bytes_sec_recv')
.as('bytes_sec_sent_avg', 'bytes_sec_recv_avg')
```

or

```
|mean('*')
.postfix('_avg')
```

would be cool.

Contributor guide

Open the contributing guide

Research direction

The issue centers on Kapacitor stream nodes such as mean, percentile, union, and influxDBOut. Start by tracing these entry points and their existing behavior; done means supporting scalable aggregation across matching fields with generated aggregate names, without requiring dozens of manually defined variables.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.