Unhandled Exceptions when /var fills up

Description of Issue

salt-master looses its ability to communicate with minion after the minion's /var hits NO SPACE.

Setup

Let /var fill up on the minion then try to test.ping it from master. After running a journalctl --vacuum-size=XXX test.ping starts working again.

Edit: Just in case this helps. In our case the minion has a service generating a lot of logs which was keeping the partition full.

Steps to Reproduce Issue

1. Fill up /var on minion:

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       100G  100G   20K 100% /var

2. Try sudo salt $minion test.ping on master

$ sudo salt MINION test.ping
MINION:
    Minion did not return. [No response]
ERROR: Minions returned with non-zero exit code

The following log is from /var/log/salt/minion on the minion after issuing a systemctl restart salt-minion

2020-01-27 09:56:08,518 [salt.utils.parsers:1071][WARNING ][2601] Minion received a SIGTERM. Exiting.
2020-01-27 09:56:14,671 [salt.utils.schedule:794 ][ERROR   ][37525] Unhandled exception running mine.update
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/schedule.py", line 693, in handle_func
    fp_.write(salt.payload.Serial(self.opts).dumps(ret))
OSError: [Errno 28] No space left on device
2020-01-27 09:56:16,641 [salt.utils.process:754 ][ERROR   ][37529] An un-handled exception from the multiprocessing process 'SignalHandlingMultiprocessingProcess-1:3' was caught:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 747, in run
    return super(MultiprocessingProcess, self).run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1594, in _target
    run_func(minion_instance, opts, data)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1589, in run_func
    return Minion._thread_return(minion_instance, opts, data)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1611, in _thread_return
    fp_.write(minion_instance.serial.dumps(sdata))
OSError: [Errno 28] No space left on device
...

Versions Report

$ salt --versions-report
Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.11 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-11-amd64
         system: Linux
        version: debian 9.11 

Start by reproducing the full-/var condition and running test.ping, then read the traceback paths in salt/utils/schedule.py, salt/utils/process.py, and salt/minion.py. Trace how OSError [Errno 28] propagates during persistence and verify that the minion handles the condition without losing communication.