influxdata / influxdata/kapacitor

Kapacitor goes OutOfMemory at regular intervals

Open
#1,994 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We have written a task that creates an alert if count is not more than zero for a period of one hour every one hour. Kapacitor service goes OutOfMemory at regular intervals. Following are the details

**Machine**
EC2 (m4.xlarge)

**Task** (has been anonymized)
```
dbrp "db"."90day"

var duration = 1h
var crit = 0

var data = stream
|from()
.measurement('line_count')
.groupBy('d_number','r_number')
|default()
.tag('type', 'old')
|where(lambda: "type" == 'device')
|window()
.period(duration)
.every(duration)
|sum('count')
.as('total_count')
|eval(lambda: int("total_count"))
.as('count_int')

var alert = data
|alert()
.id('D_Outage/{{ index .Tags "r_number" }}/{{index .Tags "d_number" }}')
.message('D {{ index .Tags "d_number"}} on LCP {{ index .Tags "r_number" }} is {{.Level}}')
.crit(lambda: "count_int" == crit)
.critReset(lambda: "count_int" > crit)

alert
.post('https://server.com/alert/kapacitor')
```
**Heap dump**
```
(pprof) top 10
Showing nodes accounting for 6645.43MB, 89.51% of 7424.30MB total
Dropped 227 nodes (cum <= 37.12MB)
Showing top 10 nodes out of 120
flat flat% sum% cum cum%
2524.86MB 34.01% 34.01% 2524.86MB 34.01% bytes.makeSlice
1230.25MB 16.57% 50.58% 1352.26MB 18.21% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.(*point).unmarshalBinary
885.74MB 11.93% 62.51% 885.74MB 11.93% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.parsePoint
630.71MB 8.50% 71.00% 630.71MB 8.50% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.Tags.Map
489.19MB 6.59% 77.59% 489.19MB 6.59% github.com/influxdata/kapacitor/tick/stateful.NewFunctions
478.18MB 6.44% 84.03% 502.18MB 6.76% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.parseTags
122MB 1.64% 85.68% 122MB 1.64% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.(*point).StringValue
95.96MB 1.29% 86.97% 95.96MB 1.29% encoding/json.(*Decoder).refill
95.52MB 1.29% 88.26% 95.52MB 1.29% github.com/influxdata/kapacitor/edge.(*pointMessage).GroupInfo
93.01MB 1.25% 89.51% 93.01MB 1.25% github.com/influxdata/kapacitor/edge.(*pointMessage).ShallowCopy
(pprof) list makeSlice
Total: 7.25GB
ROUTINE ======================== bytes.makeSlice in /usr/local/go/src/bytes/buffer.go
2.47GB 2.47GB (flat, cum) 34.01% of Total
. . 225: defer func() {
. . 226: if recover() != nil {
. . 227: panic(ErrTooLarge)
. . 228: }
. . 229: }()
2.47GB 2.47GB 230: return make([]byte, n)
. . 231:}
. . 232:
. . 233:// WriteTo writes data to w until the buffer is drained or an error occurs.
. . 234:// The return value n is the number of bytes written; it always fits into an
. . 235:// int, but it is int64 to match the io.WriterTo interface. Any error
(pprof)
```
**Kapacitor Version**
```
[root@ip-xx-xx-xx-xx ec2-user]# kapacitor version
Kapacitor OSS 1.5.0 (git: HEAD 4f10efc41b4dcac070495cf95ba2c41cfcc2aa3a)
```
**Show task output**
```
DOT:
digraph dvc_out_alert {
graph [throughput="21.00 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="940758"];

from1 [avg_exec_time_ns="31.199µs" errors="0" working_cardinality="0" ];
from1 -> default2 [processed="939757"];

default2 [avg_exec_time_ns="4.913µs" errors="0" fields_defaulted="0" tags_defaulted="0" working_cardinality="0" ];
default2 -> where3 [processed="938756"];

where3 [avg_exec_time_ns="27.573µs" errors="0" working_cardinality="50939" ];
where3 -> window4 [processed="849893"];

window4 [avg_exec_time_ns="5.107µs" errors="0" working_cardinality="49689" ];
window4 -> sum5 [processed="152363"];

sum5 [avg_exec_time_ns="10.059µs" errors="0" working_cardinality="48742" ];
sum5 -> eval6 [processed="151362"];

eval6 [avg_exec_time_ns="8.573µs" errors="0" working_cardinality="48729" ];
eval6 -> alert7 [processed="150361"];

alert7 [alerts_inhibited="0" alerts_triggered="21060" avg_exec_time_ns="729.940286ms" crits_triggered="17034" errors="0" infos_triggered="0" oks_triggered="4026" warns_triggered="0" working_cardinality="48713" ];
```

Contributor guide

Open the contributing guide

Research direction

Start with the supplied Kapacitor task and its pprof heap dump, focusing on bytes.makeSlice, models.(*point).unmarshalBinary, parsePoint, Tags.Map, and tick/stateful.NewFunctions. Reproduce the regular-interval OOM on Kapacitor OSS 1.5.0 and compare heap behavior with the reported task-output cardinalities. Done means the recurring out-of-memory condition is resolved while the task continues processing and alerting.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.