influxdata / influxdata/kapacitor

Increasing RAM usage by kapacitor

Open
#2,051 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We are using kapacitor to process our data and we see that the RAM usage considerably increases over time and does not seam to stabilize.
In our setup we use UDF's to do some complex operation and checks. With UDF's, RAM usage and build up is much more evident.

As an example I have attached example a tick script, UDF and a python script to generate the required data.
Python script - Sends 4k points to influxdb every 10 seconds. This script can be modified to send the points directly to kapacitor
Tick script - Every 10 seconds fetches last 20 seconds of data and sends it to a UDF 4 times.
UDF - Receives all the points and ignores(to simplify the testing) them

With this single tick script, kapacitor quickly starts consuming ~200MB of RAM and from there on RAM utilisation builds up slowly. In this example, UDF's does not return any points back. If it was returning RAM usage almost doubles.

When 100 such tick scripts are used Kapacitor RAM usage shoots upto ~30GB for RAM and from there on RAM utilisation builds up slowly.

[RAM.zip](https://github.com/influxdata/kapacitor/files/2361289/RAM.zip)

Could you please check if any optimisations can be done from Kapacitor side.
If there are any suggestions on the UDF side, we could incorporate them as well.

Contributor guide

Open the contributing guide

Research direction

Start with the attached RAM.zip, including the Python data generator, tick script, and UDF, and reproduce the reported memory growth with one and then 100 tick scripts. Compare Kapacitor memory usage when the UDF returns no points versus points; done means the reproduced RAM buildup is explained and a verified optimization or bounded-memory behavior is demonstrated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
data, observability-sre, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.