[BUG] Memory Leak in Saltstack 3004/3004.1 (?)

Description
We run multiple Saltstack masters in docker containers on a central platform.
Since the last update, in which we proceeded from 3003 to 3004, there seems to be a very high memory consumption, much more then before.

I saw there were changes made in 3004 to avoid memory leaks, maybe something went wrong here?
Actually I can't really tell if the behavior was introduced by 3004 or 3004.1, since we had 2 patches very close to each other, because of the breaking change in communication from 3004 to 3004.1.

Before 3004 most masters only used some hundreds of MB and I rarely saw (highly used) masters in 1-2 GB range.
Now after 3004 the seldom used masters still seem to use only "hundreds of MB" but a lot of masters go beyond 2 GB and seem to accumulate more memory over time (memory leak?), in some cases I saw up to 10GB used by a single master (before I restarted it).

The Problem seems to occur more clearly on highly/often used masters.

The Problem seems to be propagated through syndic, since it also appears on upstream-masters, even if the actual usage takes place on downstream-masters (upstream-master of test environment is actually only used for central patch-management once every 3 months, but still the issue can be seen on this master).

It can be seen (via ps aux) that the memory is consumed by saltstack master and actually a single process of it.

Setup

• Saltmasters in docker containers based on Ubuntu 20.04 (at the moment 93 masters, all of which with < 200 minions, most << 100)
• Running on Pacemaker cluster
• Minions have different OS's (Ubuntu, Debian, Centos, Redhat, Oracle Linux)

Dockerfile

FROM ubuntu:20.04

ARG SALTVERSION

RUN echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99norecommended \
 && echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/99norecommended \
 && DEBIAN_FRONTEND=noninteractive apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install ca-certificates bind9-host gnupg iproute2 iputils-ping less man-db openssh-server vim wget \
 && wget -O - https://repo.saltproject.io/py3/ubuntu/20.04/amd64/"$SALTVERSION"/salt-archive-keyring.gpg | apt-key add - \
 && echo "deb https://repo.saltproject.io/py3/ubuntu/20.04/amd64/"$SALTVERSION" focal main" >> /etc/apt/sources.list

RUN DEBIAN_FRONTEND=noninteractive apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install git python3-pip python3-git python3-setuptools salt-api salt-master salt-syndic systemd-sysv \
 && DEBIAN_FRONTEND=noninteractive apt-get autoremove \
 && DEBIAN_FRONTEND=noninteractive apt-get clean \
 && pip3 install --upgrade "CherryPy<11" salt-lint

COPY assets/* /usr/local/bin/

CMD ["/usr/local/bin/run.sh"]

Master config

fileserver_backend:
  - roots

file_roots:
  __env__:
    - /srv/salt
    - /srv/customFormulas
    - /srv/teamFormulas
    - /srv/stdFormulas

pillar_roots:
  __env__:
    - /srv/pillar
    - /srv/customPillar
    - /srv/teamPillar
    - /srv/stdPillar

reactor:
    - 'salt/minion/*/start':
        - /srv/salt/reactor/startup_sync.sls

syndic_master: 10.111.80.38
top_file_merging_strategy: same
state_output: mixed
worker_threads: 20

ps aux

(of the master cosuming >7GB in the list below)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   1084     4 ?        Ss   Apr22   0:13 /sbin/docker-init -- /usr/local/bin/run.sh
root         8  0.0  0.0   4264   580 ?        S    Apr22   0:00 sleep infinity
root      4164  0.0  0.0  12176  4368 ?        Ss   Apr22   0:00 sshd: /usr/sbin/sshd [listener] 0 of 10-10 startups
root      5793  0.0  0.0  40424  7848 ?        Ss   Apr22   0:00 /usr/sbin/sssd -D -f
root      5794  0.0  0.0 106308 23272 ?        S    Apr22   0:17 /usr/libexec/sssd/sssd_be --domain ... --uid 0 --gid 0 --logger=files
root      5795  0.0  0.1  55272 37104 ?        S    Apr22   0:04 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
root      5796  0.0  0.0  30872 11460 ?        S    Apr22   0:04 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --logger=files
root      5797  0.0  0.0  30272 11492 ?        S    Apr22   0:03 /usr/libexec/sssd/sssd_ssh --uid 0 --gid 0 --logger=files
root      5798  0.0  0.0  30544 11344 ?        S    Apr22   0:03 /usr/libexec/sssd/sssd_sudo --uid 0 --gid 0 --logger=files
root      5799  0.0  0.0  69796 15072 ?        S    Apr22   0:03 /usr/libexec/sssd/sssd_pac --uid 0 --gid 0 --logger=files
root      5827  1.2  0.0 432332  3460 ?        Ss   Apr22  75:18 gpg-agent --homedir /etc/salt/gpgkeys --use-standard-socket --daemon
root      6870  0.0  0.1 176980 59984 ?        S    Apr22   1:45 /usr/bin/python3 /usr/bin/salt-master -d
root      6871  0.1 19.4 17641024 7216300 ?    Sl   Apr22   6:56 /usr/bin/python3 /usr/bin/salt-master -d
root      6874  0.1  0.1 187068 60376 ?        S    Apr22   9:33 /usr/bin/python3 /usr/bin/salt-master -d
root      6875  0.5  0.1 993760 59108 ?        Sl   Apr22  31:17 /usr/bin/python3 /usr/bin/salt-master -d
root      6876  0.9  0.1 866516 71068 ?        S    Apr22  54:03 /usr/bin/python3 /usr/bin/salt-master -d
root      6877  0.0  0.1 169120 48928 ?        S    Apr22   0:11 /usr/bin/python3 /usr/bin/salt-master -d
root      6878  0.5  0.1 472144 67424 ?        Sl   Apr22  29:42 /usr/bin/python3 /usr/bin/salt-master -d
root      6885 12.8  0.2 888800 79280 ?        Rl   Apr22 765:16 /usr/bin/python3 /usr/bin/salt-master -d
root      6886 12.8  0.2 880856 79396 ?        Sl   Apr22 763:05 /usr/bin/python3 /usr/bin/salt-master -d
root      6888 12.9  0.2 883300 81820 ?        Sl   Apr22 765:48 /usr/bin/python3 /usr/bin/salt-master -d
root      6890 12.8  0.2 883532 82028 ?        Rl   Apr22 762:43 /usr/bin/python3 /usr/bin/salt-master -d
root      6893 12.8  0.2 887660 77084 ?        Sl   Apr22 765:25 /usr/bin/python3 /usr/bin/salt-master -d
root      9673  0.7  0.1 176980 57884 ?        Sl   11:47   0:01 /usr/bin/python3 /usr/bin/salt-master -d
root     11578  0.0  0.0   7648  3340 pts/0    R+   11:51   0:00 ps aux

Some containers from docker stats

CONTAINER	MEM
f547949534ce	769.5MiB
4aaa3ab6d9de	993.7MiB
6fe9d7a71e30	1.086GiB
dc862524e87a	7.437GiB
d5a96aba8e8e	668.4MiB
422423ed36bd	723.7MiB
d54fbc52afed	644.1MiB
94391b8ac74e	650.1MiB
7c4b6dd48b1d	668.9MiB
0a94d8839db9	737.4MiB
aa762b99dd65	614MiB
cf3372821f82	439.3MiB
925858cf96ab	707.2MiB
23f1590cb0b5	933.5MiB
43372d3f4277	453.9MiB
9f6e519b0c88	456.3MiB
d77cef3c69f1	983.6MiB
17af3909b82e	496.6MiB
0b874b5689fa	706.4MiB
9c056d9b2383	1.198GiB
20e63edca7d0	478MiB
435cbfb77722	470.1MiB
cc268e1fd1a3	468.7MiB
346ecbd901cd	480.4MiB
2a44dfd46643	468.7MiB
bb15cfafcdfb	471.6MiB
c43e57fc8007	472.3MiB
e16f66097b50	3.29GiB
2cea5adceeb3	804.8MiB
6d487e1d16ef	2.283GiB
aa8929509467	2.344GiB
27eac559c1fe	2.544GiB
aaad4b8fe0e0	2.983GiB
2ef823f5641d	2.592GiB
45b906badd18	793.7MiB
d2013cda9e81	468.4MiB
d02777c79c45	407.4MiB
e37746b23bcf	434.2MiB
092b867b3cb1	479.7MiB
30e49e22f148	473.3MiB
b91eed7ae140	2.47GiB
85cc03300931	2.503GiB
df48cde421e6	428.9MiB
a11fe5bd7433	2.073GiB
cae21af8d292	2.169GiB
87f6b996d1b8	1.103GiB
5b2bf0e53b9d	1.331GiB
38ea4477113e	1.29GiB
8f71852ee073	685.5MiB
48127b4aabc8	543MiB

Versions Report

Salt Version:
          Salt: 3004.1
 
Dependency Versions:
          cffi: 1.14.0
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.19
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.10 (default, Mar 15 2022, 12:22:08)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 4.15.0-171-generic
        system: Linux
       version: Ubuntu 20.04 focal