Potential Memory Leak in Argo Workflows

### Pre-requisites

- [X] I have double-checked my configuration
- [X] I can confirm the issues exists when I tested with `:latest`
- [ ] I'd like to contribute the fix myself (see [contributing guide](https://github.com/argoproj/argo-workflows/blob/master/docs/CONTRIBUTING.md))

### What happened/what you expected to happen?

I've just been investigating https://github.com/argoproj/argo-workflows/issues/10219.
In order to be sure there weren't any Argo Workflows related memory issues, I decided to test on large workflows.
In addition to the workflow being large, I ran it multiple times. When running these large workflows, I was able to continuously push the memory usage by the controller up and up.

I'm not sure if there is an upper bound on this, I think I'd need a more powerful computer to test this to the 100+ workflow runs to check if there is an upper bound. I think having an upper bound is the desired outcome here, completely unbounded memory growth maybe the issue that is taking controllers down on extremely large workflows. I think I peaked at about 120M

Here are the screenshots of memory usage in chronological order:

### Version

latest

### Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

```YAML
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dag-diamond-steps-
spec:
entrypoint: diamond
templates:
- name: echo
inputs:
parameters:
- name: message
container:
image: alpine:3.7
command: [echo, "{{inputs.parameters.message}}"]
- name: echo-thrice
inputs:
parameters:
- name: message
steps:
- - name: echo
template: echo
arguments:
parameters:
- {name: message, value: "{{inputs.parameters.message}}{{item}}"}
withItems: [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,89,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100]
- name: diamond
dag:
tasks:
- name: A
template: echo-thrice
arguments:
parameters: [{name: message, value: A}]
- name: B
depends: "A"
template: echo-thrice
arguments:
parameters: [{name: message, value: B}]
- name: C
depends: "A"
template: echo-thrice
arguments:
parameters: [{name: message, value: C}]
- name: D
depends: "B && C"
template: echo-thrice
arguments:
parameters: [{name: message, value: D}]
```

### Logs from the workflow controller

```text
N/A
```

### Logs from in your workflow's wait container

```text
N/A
```