Huge memory usage increase in v0.19.1+

**Which component**:
Controller v0.19.1+
(didn't checked v0.19.0 since it doesn't have published chart

**Describe the bug**
After update from any previous version memory usage from ~7mb to ~50+ mb (slightly drops after some time)

**To Reproduce**
Steps to reproduce the behavior:

1. Update from v0.18.5 to v0.19.1
2. Check graph on memory usage of container

**Expected behavior**
Memory usage not increase so drastically

**Version of Kubernetes**:

- Output of `kubectl version`:

```
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25+", GitVersion:"v1.25.12-eks-2d98532", GitCommit:"0aa16cf4fac4da27b9e9e9ba570b990867f6a3d8", GitTreeState:"clean", BuildDate:"2023-07-28T16:52:04Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
```

**Additional context**
Basically we're running controllers with 20MB limits/requests in the kubernetes.
It was OK for a long time, but new version went to OOMKilled and only runs now with at least 60MB or more

While that's not a big numbers at all, it's still strange that there's so big memory usage spike without getting some cool additional features.

Feel free to close the issue though since it's mostly just a rant 😄

Start by comparing the Controller deployments and memory metrics for v0.18.5 and v0.19.1, using the reported Kubernetes environment and 20 MB limit as the baseline. Trace the change responsible for the increase and verify the fix by reproducing the upgrade without the controller being OOMKilled or requiring at least 60 MB.

Written by the indexing model from the issue text.