gpg decryption fails with large number of SSH connections

Description of Issue/Question

When I'm applying states using salt-ssh with a big enough roster (53 machines in my case), and default max_procs (25), I usually get several Failed to return clean data errors in the beginning of the execution. Recently I started to put encrypted data in pillar, and now I get gpg decryption errors:

$ salt-ssh '*' state.apply
server1:
    ----------
    _error:
        Failed to return clean data
    retcode:
        255
    stderr:
        ssh_exchange_identification: Connection closed by remote host
        ssh_exchange_identification: Connection closed by remote host
    stdout:
server1:
    ----------
    _error:
        Failed to return clean data
    retcode:
        255
    stderr:
        ssh_exchange_identification: Connection closed by remote host
        ssh_exchange_identification: Connection closed by remote host
    stdout:

[WARNING ] Could not decrypt cipher -----BEGIN PGP MESSAGE-----

.....
-----END PGP MESSAGE-----
, received: [GNUPG:] ENC_TO .....
[GNUPG:] KEY_CONSIDERED .....
[GNUPG:] KEY_CONSIDERED .....
gpg: encrypted with 2048-bit RSA key, ID ....., created 2019-01-17
      "...."
gpg: public key decryption failed: Cannot allocate memory
[GNUPG:] ERROR pkdecrypt_failed 16810070
[GNUPG:] BEGIN_DECRYPTION
[GNUPG:] DECRYPTION_FAILED
gpg: decryption failed: No secret key
[GNUPG:] END_DECRYPTION

While connection usually fails only for few targets, decryption error occurs for all of them, and state execution continues after that, so target machines receive encrypted data instead of decrypted. For example, instead of the password some config will contain encrypted message.

It is not clear whether decryption errors are related to connection errors or not. But both occur only with large number of workers, and could be prevented by setting max_procs to lower value such as 8. There are Cannot allocate memory messages in the log, but memory is not exhausted.

Setup

pillar:

#!yaml|gpg

smtp:
  password: |
    -----BEGIN PGP MESSAGE-----
    .....
    -----END PGP MESSAGE-----

state:

test_state:
   cmd.run:
    - name: 'echo "{{ pillar['smtp']['password'] }}"'

Versions Report

$ salt-ssh --versions-report
Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 2.7.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.30.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.0
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.7.3
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15 (default, Jan 10 2019, 23:20:52)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 5.1.1
            ZMQ: 4.3.1
 
System Versions:
           dist:   
         locale: UTF-8
        machine: x86_64
        release: 4.20.3-arch1-1-ARCH
         system: Linux
        version: Not Installed
 

Start with the salt-ssh state.apply entry point and reproduce the issue using a roster of about 53 machines with max_procs set to 25, then compare it with max_procs set to 8. Examine the connection-closed and GPG Cannot allocate memory errors together; done means encrypted pillar values are decrypted correctly and targets no longer receive ciphertext during large concurrent runs.