influxdata / influxdata/kapacitor

Unexpected high system memory usage on single stream task

Open
#1,586 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi,

I'm working with Kapacitor 1.3.1 (git: master 3b5512f) to create some alerts with an InfluxDB subscription and after working on databases and several TickScripts we noticed an unusual high memory usage.

We have only enabled one task predefined with a template and filled with JSON which is retrieving data from `stream` mode to:

- Calculate the mean `every 1m` of `30m windowed` of `InputUtilization` in every `interface` of every `switch`.
- Total interfaces of switches reaches to 4024.
- Some of the interfaces may not have the field `InputUtilization`, so, sometimes error on mean nodes could appear.

## Example case

After enabling the task and see that it works OK, we have seen that the used system memory increased to unexpected level since we started Kapacitor and it reaches a constant level after 1 hour of working.

````
Memory before: 1.907 GB
Memory after 1 hour: 2.985 GB
Total increase 1.078 GB
````

![memory increase](https://user-images.githubusercontent.com/13196237/30860455-6e57253a-a2c7-11e7-8bd6-d9847f40ee65.jpg)

- Assuming it keeps the data for 30min (window .period()) `30 min * 4024 series * 1 point/1min = 120720 points over 30 min of data`, it seems each point is using `1,078GB / 120720 points = 9588 bytes`. We think this is too much memory and we can not understand why Kapacitor requires this amount of memory for each point.

- We don't understand why the used system memory reaches the max level after 1 hour if only windowing points with interval set up with 30m duration.

## Questions

With the behaviour explained above:

1. Does kapacitor need 10Kb of data to store each point of data on stream mode ?

2. Assuming linear scaling and the same cardinality, for only `N` task, we would need `N * 1GB` of memory?

3. Could you help us to understand Kapacitor memory behaviour, please ?

We attach you the following TickScripts and show statistics to let you check if there is something wrong on our TickScript/Nodes definition:

## TickScript + node info

````
// TICKSCRIPT:
// ================
var data = stream
|from()
.database('switch_metrics')
.retentionPolicy('1y')
.measurement('ifMIB_metrics')
.groupBy([*])
.where(lambda: TRUE)
|window()
.period(30m)
.every(1m)
.align()
|mean('InputUtilization')
.as('value')

// check_alert generates a 'mon_check' field that brief if a device has to be monitored or not
var check_alert = data
|default()
.field('mon_active', TRUE)
.field('mon_line', 'LBLE')
.field('mon_exc', 0)
.tag('poc_entorno', 'PRO')
// Generate hour and day
|eval(lambda: hour(time), lambda: weekday(time))
.as('time_hour', 'time_day')
.keep()
|eval(lambda: if(mon_activo == TRUE AND mon_exc >= 0 AND strContains(mon_line, ID_LINIA), 1, 0))
.as('mon_check')
.keep()
|httpOut('mon_check')

var trigger = check_alert
|alert()
.crit(lambda: if(mon_check == 1 AND time_hour >= TH_CRIT_MIN_HOUR AND time_hour <= TH_CRIT_MAX_HOUR AND strContains(WEEK_DAY_CRIT, string(time_day)), float(value) > if(mon_exc == 0, TH_CRIT_DEF, if(mon_exc == 1, TH_CRIT_EX1, if(mon_exc == 2, TH_CRIT_EX2, 0.0))), FALSE))
.warn(lambda: if(mon_check == 1 AND time_hour >= TH_WARN_MIN_HOUR AND time_hour <= TH_WARN_MAX_HOUR AND strContains(WEEK_DAY_WARN, string(time_day)), float(value) > if(mon_exc == 0, TH_WARN_DEF, if(mon_exc == 1, TH_WARN_EX1, if(mon_exc == 2, TH_WARN_EX2, 0.0))), FALSE))
.info(lambda: if(mon_check == 1 AND time_hour >= TH_INFO_MIN_HOUR AND time_hour <= TH_INFO_MAX_HOUR AND strContains(WEEK_DAY_INFO, string(time_day)), float(value) > if(mon_exc == 0, TH_INFO_DEF, if(mon_exc == 1, TH_INFO_EX1, if(mon_exc == 2, TH_INFO_EX2, 0.0))), FALSE))
.stateChangesOnly()
.id(ID)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.message(message)
.details(details)
.email()
.to(['foo@mail.com','bar@mail.com'])
trigger
|httpOut('output')
````

The following table is the brief of nodes cardinality and the diagram:
Note: Sometimes there is an error on the `mean node` because the field on the point does not exist.

| Node | exc_time | errs | cardin.| processed |
|------|------------------|--------|-------- | --------- |
stream0 |0s |0 |0 | stream0 -> from1
from1 |35.568µs |0 |0 |from1 -> window2
window2 |16.249µs |0 |4024 |window2 -> mean3
mean3 |81.9µs |91 |0 |mean3 -> default4
default4 |43.656µs |0 |0 | default4->eval5
eval5 |343.215µs |0 |4024 |eval5 -> eval6
eval6 |134.317µs |0 |4024 | eval6 -> http_out7
http_out7 |24.021µs |0 |4024 | http_out7 -> alert8
alert8 | 158.438µs |0 |4024 |alert8 -> http_out9
http_out9 |0s |0 |0 |

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported memory growth with the supplied TickScript, especially the 30m window, 1m interval, and 4024-series cardinality. Inspect the window and mean node behavior and compare memory before and after an hour. Done means explaining the observed memory use and identifying a bounded-memory fix or confirming that the behavior is expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.