salt-ssh generates error deploying custom grain on first run, works on subsequent runs

Description of Issue

I've run into a real weird one that I'm not sure what's going on. Basically, any time I add or change one of my custom modules and then run salt-ssh, it errors out with the following:

   _error:
        Failed to return clean data
    retcode:
        0
    stderr:
        Shared connection to 10.1.15.93 closed.
    stdout:
        ERROR: Failure deploying ext_mods: /usr/bin/scp
        SALT_ARGV: ['/usr/bin/python3', '/var/tmp/.ubuntu_c08594_salt/salt-call', '--retcode-passthrough', '--local', '--metadata', '--out', 'json', '-l', 'quiet', '-c', '/var/tmp/.ubuntu_c08594_salt', '--', 'test.opts_pkg']
[... cut massive debug output with lots of private info ...]

however, future runs against the same host work correctly with the custom grain set. so it seems like the deploy of the custom grain did work, error message aside. still, it means I can't trust any salt-ssh run to work correctly the first time.

Setup

I can trigger this by adding or modifying any custom grain, using salt installed under python3 via pip. Not sure what out of my environment is useful, this seems to be salt code problem, but I guess that's only the case if y'all can reproduce.

in any case, I'm using salt without a server and primarily with salt-ssh as a non-root user, so here's the relevant custom configs.

~/.salt/Saltfile:

salt-call:
  config_dir: /home/tessa/.salt/conf/
salt-ssh:
  config_dir: /home/tessa/.salt/conf/
  # don't clean up temp python files on remote host when done,
  # will make subsequent ops faster.
  ssh_wipe: False

~/.salt/conf/minion

# set a fixed minion id so we can share a top.sls between salt-call and salt-ssh
id: local

ext_pillar:
  - ec2_pillar:
      tag_list_key:
        - salt_states
      tag_list_sep: ','
      use_grain: True

file_client: local
# all of this 
file_roots:
  base:
    - /home/tessa/.salt
    - /home/tessa/.salt/_states
log_level: error
module_dirs:
  - /home/tessa/.salt
pillar_roots:
  base:
    - /home/tessa/.salt/_pillar
root_dir: /tmp/.salt-root


# salt-ssh options end up here
roster: ec2
roster_defaults:
  # put config for remote hosts here!
  minion_opts:
    grains_cache: False
    metadata_server_grains: True
  priv: agent-forwarding
  sudo: True
  tty: True
  user: ubuntu

# only show changes by default
state_output: changes

# any python libs our custom modules depend on must be included here
# or they won't be bundled to the client.
thin_extra_mods: boto,boto3

Versions Report

Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: 1.12.3
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Aug 20 2019, 17:04:43)
   python-gnupg: Not Installed
         PyYAML: 5.1
          PyZMQ: 18.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1
 
System Versions:
           dist: Ubuntu 19.04 disco
         locale: UTF-8
        machine: x86_64
        release: 5.0.0-25-generic
         system: Linux
        version: Ubuntu 19.04 disco

Start with the salt-ssh first-run deployment path for custom grains and the reported “Failure deploying ext_mods” output. Reproduce with a custom module or grain using the shown Saltfile and minion configuration, then compare the first and subsequent runs. Done means the initial run completes without the deployment error and still applies the custom grain.