influxdata / influxdata/kapacitor
Aggregating data with multiple fields
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I'm trying to aggregate all fields of a certain series to time steps of 1h. With continuous queries, this worked as follows:
```
CREATE CONTINUOUS QUERY cq_1h_net_sec ON telegraf
BEGIN
SELECT
mean(bytes_sec_recv) as bytes_sec_recv_avg,
max(bytes_sec_recv) as bytes_sec_recv_max,
min(bytes_sec_recv) as bytes_sec_recv_min,
percentile(bytes_sec_recv, 95) as bytes_sec_recv_perc95,
mean(bytes_sec_sent) as bytes_sec_sent_avg,
max(bytes_sec_sent) as bytes_sec_sent_max,
min(bytes_sec_sent) as bytes_sec_sent_min
percentile(bytes_sec_sent, 95) as bytes_sec_sent_perc95,
INTO "store1y"."net_sec"
FROM "store7d".net_sec
GROUP BY *, time(1h)
END
```
In the example, I process two fields, but I'm searching for a more general solution that processes all fields, a regex match or the like. With kapacitor, I don't see a solution yet; maybe splitting, processing and joining fields?
```
var val_raw = stream
|from()
.database('telegraf')
.retentionPolicy('store7d')
.measurement('net_sec')
.groupBy(*)
|default()
.field('bytes_sec_recv', 0)
.field('bytes_sec_sent', 0)
var windowed_vals = val_raw
|window()
.period(30s)
.every(30s)
.align()
var bytes_sec_recv_avg = windowed_vals
|mean('bytes_sec_recv')
.as('bytes_sec_recv_avg')
var bytes_sec_sent_avg = windowed_vals
|mean('bytes_sec_sent')
.as('bytes_sec_sent_avg')
bytes_sec_recv_avg
|union(bytes_sec_sent_avg)
|influxDBOut()
.database('telegraf')
.retentionPolicy('store1y')
.measurement('net_sec')
.precision('s')
```
Is this the way to go? I'm not sure if this scales well, as there are ~8 fields (rx and tx of bytes, packets, errors and drops) and 6 aggregation functions (min, max, avg, percentile (25, 75, 95)) per telegraf net series, leading to 48 variables...
Something like
```
|mean('bytes_sec_sent', 'bytes_sec_recv')
.as('bytes_sec_sent_avg', 'bytes_sec_recv_avg')
```
or
```
|mean('*')
.postfix('_avg')
```
would be cool.
Contributor guide
Research direction
The issue centers on Kapacitor stream nodes such as mean, percentile, union, and influxDBOut. Start by tracing these entry points and their existing behavior; done means supporting scalable aggregation across matching fields with generated aggregate names, without requiring dozens of manually defined variables.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100