influxdata / influxdata/kapacitor
Kapacitor goes OutOfMemory at regular intervals
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We have written a task that creates an alert if count is not more than zero for a period of one hour every one hour. Kapacitor service goes OutOfMemory at regular intervals. Following are the details
**Machine**
EC2 (m4.xlarge)
**Task** (has been anonymized)
```
dbrp "db"."90day"
var duration = 1h
var crit = 0
var data = stream
|from()
.measurement('line_count')
.groupBy('d_number','r_number')
|default()
.tag('type', 'old')
|where(lambda: "type" == 'device')
|window()
.period(duration)
.every(duration)
|sum('count')
.as('total_count')
|eval(lambda: int("total_count"))
.as('count_int')
var alert = data
|alert()
.id('D_Outage/{{ index .Tags "r_number" }}/{{index .Tags "d_number" }}')
.message('D {{ index .Tags "d_number"}} on LCP {{ index .Tags "r_number" }} is {{.Level}}')
.crit(lambda: "count_int" == crit)
.critReset(lambda: "count_int" > crit)
alert
.post('https://server.com/alert/kapacitor')
```
**Heap dump**
```
(pprof) top 10
Showing nodes accounting for 6645.43MB, 89.51% of 7424.30MB total
Dropped 227 nodes (cum <= 37.12MB)
Showing top 10 nodes out of 120
flat flat% sum% cum cum%
2524.86MB 34.01% 34.01% 2524.86MB 34.01% bytes.makeSlice
1230.25MB 16.57% 50.58% 1352.26MB 18.21% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.(*point).unmarshalBinary
885.74MB 11.93% 62.51% 885.74MB 11.93% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.parsePoint
630.71MB 8.50% 71.00% 630.71MB 8.50% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.Tags.Map
489.19MB 6.59% 77.59% 489.19MB 6.59% github.com/influxdata/kapacitor/tick/stateful.NewFunctions
478.18MB 6.44% 84.03% 502.18MB 6.76% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.parseTags
122MB 1.64% 85.68% 122MB 1.64% github.com/influxdata/kapacitor/vendor/github.com/influxdata/influxdb/models.(*point).StringValue
95.96MB 1.29% 86.97% 95.96MB 1.29% encoding/json.(*Decoder).refill
95.52MB 1.29% 88.26% 95.52MB 1.29% github.com/influxdata/kapacitor/edge.(*pointMessage).GroupInfo
93.01MB 1.25% 89.51% 93.01MB 1.25% github.com/influxdata/kapacitor/edge.(*pointMessage).ShallowCopy
(pprof) list makeSlice
Total: 7.25GB
ROUTINE ======================== bytes.makeSlice in /usr/local/go/src/bytes/buffer.go
2.47GB 2.47GB (flat, cum) 34.01% of Total
. . 225: defer func() {
. . 226: if recover() != nil {
. . 227: panic(ErrTooLarge)
. . 228: }
. . 229: }()
2.47GB 2.47GB 230: return make([]byte, n)
. . 231:}
. . 232:
. . 233:// WriteTo writes data to w until the buffer is drained or an error occurs.
. . 234:// The return value n is the number of bytes written; it always fits into an
. . 235:// int, but it is int64 to match the io.WriterTo interface. Any error
(pprof)
```
**Kapacitor Version**
```
[root@ip-xx-xx-xx-xx ec2-user]# kapacitor version
Kapacitor OSS 1.5.0 (git: HEAD 4f10efc41b4dcac070495cf95ba2c41cfcc2aa3a)
```
**Show task output**
```
DOT:
digraph dvc_out_alert {
graph [throughput="21.00 points/s"];
stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="940758"];
from1 [avg_exec_time_ns="31.199µs" errors="0" working_cardinality="0" ];
from1 -> default2 [processed="939757"];
default2 [avg_exec_time_ns="4.913µs" errors="0" fields_defaulted="0" tags_defaulted="0" working_cardinality="0" ];
default2 -> where3 [processed="938756"];
where3 [avg_exec_time_ns="27.573µs" errors="0" working_cardinality="50939" ];
where3 -> window4 [processed="849893"];
window4 [avg_exec_time_ns="5.107µs" errors="0" working_cardinality="49689" ];
window4 -> sum5 [processed="152363"];
sum5 [avg_exec_time_ns="10.059µs" errors="0" working_cardinality="48742" ];
sum5 -> eval6 [processed="151362"];
eval6 [avg_exec_time_ns="8.573µs" errors="0" working_cardinality="48729" ];
eval6 -> alert7 [processed="150361"];
alert7 [alerts_inhibited="0" alerts_triggered="21060" avg_exec_time_ns="729.940286ms" crits_triggered="17034" errors="0" infos_triggered="0" oks_triggered="4026" warns_triggered="0" working_cardinality="48713" ];
```
Contributor guide
Research direction
Start with the supplied Kapacitor task and its pprof heap dump, focusing on bytes.makeSlice, models.(*point).unmarshalBinary, parsePoint, Tags.Map, and tick/stateful.NewFunctions. Reproduce the regular-interval OOM on Kapacitor OSS 1.5.0 and compare heap behavior with the reported task-output cardinalities. Done means the recurring out-of-memory condition is resolved while the task continues processing and alerting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100