Chains under the load memory consumption raises

Expected Behavior

I would expect after some time Chains under the load memory consumption would become constant - it will start freeing memory.

Is this expected, or is this some sort of memory leak?

Actual Behavior

This is a memory graph of a Chains signing 10k very simple TaskRuns with that are just printing "hello world" (Pipeline, PipelineRun)

CHains was started around 15:30 and started signing PRs and TRs:

Chains was configured with this:

kubectl patch TektonConfig/config \
            --type merge \
            -p '{"spec":{"chain":{"artifacts.pipelinerun.format": "slsa/v1"}}}'
kubectl patch TektonConfig/config \
            --type merge \
            -p '{"spec":{"chain":{"artifacts.pipelinerun.storage": "tekton"}}}'
kubectl patch TektonConfig/config \
            --type='merge' \
            -p='{"spec":{"chain":{"artifacts.taskrun.format": "slsa/v1"}}}'
kubectl patch TektonConfig/config \
            --type='merge' \
            -p='{"spec":{"chain":{"artifacts.taskrun.storage": "tekton"}}}'

Steps to Reproduce the Problem

1. Run 10k PipelineRuns and wait for all of them to finish
2. Then start the Chains and let it sign PRs and TRs
3. This was automated in this repo with signing-tr-tekton-bigbang scenario

Additional Info

• Kubernetes version:

Cluster is gone already, but it was ROSA OpenShift 4.14.11 with 5 compute nodes AWS EC2 m6a.2xlarge

• Tekton Pipeline version:

• chains-info: v0.19.0
• pipelines-as-code-info: v0.22.4
• pipelines-info: v0.53.2
• triggers-info: v0.25.3
• openshift-pipelines-operator-cluster-operations: v0.69.0
• openshift-pipelines-operator-lifecycle: 1.13.0

Reported this together with https://github.com/tektoncd/pipeline/issues/7691

Start with the linked performance scenario in pipeline.yaml, run.yaml, and its README, then reproduce the 10k PipelineRun signing workload described here. Review the related tektoncd/pipeline issue 7691 and the Chains implementation involved in signing. Done means identifying whether the memory growth is expected or a leak, with a validated fix or clear evidence and regression coverage.