Frontend Service - goroutine (CPU & Memory) Leak

Expected Behavior

There should be no memory leak resulting from objects not being properly garbage collected.

Actual Behavior

Number of objects on Heap keeps growing. This seems to be result in slow increase of cpu & memory usage eventually resulting in outage.

Steps to Reproduce the Problem

• Start server

temporal server start-dev --port 7233 --ui-port 8233 --metrics-port 9233

• Do not start workflows or make any grpc calls, use the sdks or the web ui. aka keep the usage to a minimum.

• Periodically check pprof & metrics. This shows that the goroutinue counts, objects on heap (aka memory allocations for objects) & memory allocations keeps growing perpetually.

• runtime.MemStats from from http://localhost:/debug/pprof/heap?debug=1

# runtime.MemStats
# Alloc = 65091800
# TotalAlloc = 318383929504
# Sys = 119521560
# Lookups = 0
# Mallocs = 1641446565
# Frees = 1641167975
# HeapAlloc = 65091800
# HeapSys = 82378752
# HeapIdle = 11141120
# HeapInuse = 71237632
# HeapReleased = 4464640
# HeapObjects = 278590
# Stack = 18284544 / 18284544
# MSpan = 825408 / 960048
# MCache = 19200 / 31200
# BuckHashSys = 2638467
# GCSys = 12561120
# OtherSys = 2667429
# NextGC = 69589984
# LastGC = 1721688648441657922
...
# NumGC = 10489
# NumForcedGC = 0
# GCCPUFraction = 4.835076125703521e-05
# DebugGC = false
# MaxRSS = 178909184

• Flame graph

• This was observed across local & shared & production environments. Please see the prometheus chart in a production environment where the num_goroutine count kept increasing until a restart. Notice that the "leap" appears to be isolated to the frontend service. The rest seem fine. The CPU usage & memory usage charts followed the same pattern.

Specifications

This was observed across multiple versions

• Server Version: 1.22.5, 1.22.7, 1.23.1, etc.
• Platform: Linux
• MTLS enabled
• Auth disabled

Links

This issue is potentially related to https://community.temporal.io/t/high-cpu-usage-memory-leakage-on-frontend-service/4246/1