influxdata / influxdata/kapacitor
Unexpected high system memory usage on single stream task
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi,
I'm working with Kapacitor 1.3.1 (git: master 3b5512f) to create some alerts with an InfluxDB subscription and after working on databases and several TickScripts we noticed an unusual high memory usage.
We have only enabled one task predefined with a template and filled with JSON which is retrieving data from `stream` mode to:
- Calculate the mean `every 1m` of `30m windowed` of `InputUtilization` in every `interface` of every `switch`.
- Total interfaces of switches reaches to 4024.
- Some of the interfaces may not have the field `InputUtilization`, so, sometimes error on mean nodes could appear.
## Example case
After enabling the task and see that it works OK, we have seen that the used system memory increased to unexpected level since we started Kapacitor and it reaches a constant level after 1 hour of working.
````
Memory before: 1.907 GB
Memory after 1 hour: 2.985 GB
Total increase 1.078 GB
````

- Assuming it keeps the data for 30min (window .period()) `30 min * 4024 series * 1 point/1min = 120720 points over 30 min of data`, it seems each point is using `1,078GB / 120720 points = 9588 bytes`. We think this is too much memory and we can not understand why Kapacitor requires this amount of memory for each point.
- We don't understand why the used system memory reaches the max level after 1 hour if only windowing points with interval set up with 30m duration.
## Questions
With the behaviour explained above:
1. Does kapacitor need 10Kb of data to store each point of data on stream mode ?
2. Assuming linear scaling and the same cardinality, for only `N` task, we would need `N * 1GB` of memory?
3. Could you help us to understand Kapacitor memory behaviour, please ?
We attach you the following TickScripts and show statistics to let you check if there is something wrong on our TickScript/Nodes definition:
## TickScript + node info
````
// TICKSCRIPT:
// ================
var data = stream
|from()
.database('switch_metrics')
.retentionPolicy('1y')
.measurement('ifMIB_metrics')
.groupBy([*])
.where(lambda: TRUE)
|window()
.period(30m)
.every(1m)
.align()
|mean('InputUtilization')
.as('value')
// check_alert generates a 'mon_check' field that brief if a device has to be monitored or not
var check_alert = data
|default()
.field('mon_active', TRUE)
.field('mon_line', 'LBLE')
.field('mon_exc', 0)
.tag('poc_entorno', 'PRO')
// Generate hour and day
|eval(lambda: hour(time), lambda: weekday(time))
.as('time_hour', 'time_day')
.keep()
|eval(lambda: if(mon_activo == TRUE AND mon_exc >= 0 AND strContains(mon_line, ID_LINIA), 1, 0))
.as('mon_check')
.keep()
|httpOut('mon_check')
var trigger = check_alert
|alert()
.crit(lambda: if(mon_check == 1 AND time_hour >= TH_CRIT_MIN_HOUR AND time_hour <= TH_CRIT_MAX_HOUR AND strContains(WEEK_DAY_CRIT, string(time_day)), float(value) > if(mon_exc == 0, TH_CRIT_DEF, if(mon_exc == 1, TH_CRIT_EX1, if(mon_exc == 2, TH_CRIT_EX2, 0.0))), FALSE))
.warn(lambda: if(mon_check == 1 AND time_hour >= TH_WARN_MIN_HOUR AND time_hour <= TH_WARN_MAX_HOUR AND strContains(WEEK_DAY_WARN, string(time_day)), float(value) > if(mon_exc == 0, TH_WARN_DEF, if(mon_exc == 1, TH_WARN_EX1, if(mon_exc == 2, TH_WARN_EX2, 0.0))), FALSE))
.info(lambda: if(mon_check == 1 AND time_hour >= TH_INFO_MIN_HOUR AND time_hour <= TH_INFO_MAX_HOUR AND strContains(WEEK_DAY_INFO, string(time_day)), float(value) > if(mon_exc == 0, TH_INFO_DEF, if(mon_exc == 1, TH_INFO_EX1, if(mon_exc == 2, TH_INFO_EX2, 0.0))), FALSE))
.stateChangesOnly()
.id(ID)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.message(message)
.details(details)
.email()
.to(['foo@mail.com','bar@mail.com'])
trigger
|httpOut('output')
````
The following table is the brief of nodes cardinality and the diagram:
Note: Sometimes there is an error on the `mean node` because the field on the point does not exist.
| Node | exc_time | errs | cardin.| processed |
|------|------------------|--------|-------- | --------- |
stream0 |0s |0 |0 | stream0 -> from1
from1 |35.568µs |0 |0 |from1 -> window2
window2 |16.249µs |0 |4024 |window2 -> mean3
mean3 |81.9µs |91 |0 |mean3 -> default4
default4 |43.656µs |0 |0 | default4->eval5
eval5 |343.215µs |0 |4024 |eval5 -> eval6
eval6 |134.317µs |0 |4024 | eval6 -> http_out7
http_out7 |24.021µs |0 |4024 | http_out7 -> alert8
alert8 | 158.438µs |0 |4024 |alert8 -> http_out9
http_out9 |0s |0 |0 |
Contributor guide
Research direction
Start by reproducing the reported memory growth with the supplied TickScript, especially the 30m window, 1m interval, and 4024-series cardinality. Inspect the window and mean node behavior and compare memory before and after an hour. Done means explaining the observed memory use and identifying a bounded-memory fix or confirming that the behavior is expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100