influxdata / influxdata/kapacitor
Increasing RAM usage by kapacitor
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We are using kapacitor to process our data and we see that the RAM usage considerably increases over time and does not seam to stabilize.
In our setup we use UDF's to do some complex operation and checks. With UDF's, RAM usage and build up is much more evident.
As an example I have attached example a tick script, UDF and a python script to generate the required data.
Python script - Sends 4k points to influxdb every 10 seconds. This script can be modified to send the points directly to kapacitor
Tick script - Every 10 seconds fetches last 20 seconds of data and sends it to a UDF 4 times.
UDF - Receives all the points and ignores(to simplify the testing) them
With this single tick script, kapacitor quickly starts consuming ~200MB of RAM and from there on RAM utilisation builds up slowly. In this example, UDF's does not return any points back. If it was returning RAM usage almost doubles.
When 100 such tick scripts are used Kapacitor RAM usage shoots upto ~30GB for RAM and from there on RAM utilisation builds up slowly.
[RAM.zip](https://github.com/influxdata/kapacitor/files/2361289/RAM.zip)
Could you please check if any optimisations can be done from Kapacitor side.
If there are any suggestions on the UDF side, we could incorporate them as well.
Contributor guide
Research direction
Start with the attached RAM.zip, including the Python data generator, tick script, and UDF, and reproduce the reported memory growth with one and then 100 tick scripts. Compare Kapacitor memory usage when the UDF returns no points versus points; done means the reproduced RAM buildup is explained and a verified optimization or bounded-memory behavior is demonstrated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- data, observability-sre, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100