salt-ssh: Failed to return clean data/invalid opcodes on minions with old CPU

Hello,

Description of issue

I'm using salt-ssh 2019.2.2 on Gentoo Linux with a Coffee Lake CPU and Debian8/9/10 minions with various Intel CPUs. All architectures are amd64.

On minions with older CPUs, such as Ivy Bridge, I've recently started encountering this problem:

    _error:
        Failed to return clean data
    retcode:
        252

On the minion I can see logs such as:

kernel: traps: python2.7[108098] trap invalid opcode ip:7f734a2cf15a sp:7ffe7d5f9820 error:0 in _cmsgpack.so[7f734a2bb000+19000]

Setup

I suspect that because my host system is Gentoo Linux, where compiling packages with -march=native is common, thus creating binaries that might not run on older CPUs. This is normally not a problem. My theory is that this _cmsgpack.so is deployed onto the minion and is not executable, resulting in the error above.

However various factors make this less than clear:

1. While salt-ssh reads the host's /usr/lib64/python2.7/site-packages/msgpack/_cmsgpack.so (verified with strace), the file that ends up being deployed to the minion under /var/tmp/.salt-*/pyall/msgpack/_cmsgpack.so is not the same (different size and MD5)
2. Clearing minion's cache in /var/tmp only makes salt redeploy it with the borked version
3. Even removing _cmsgpack.so from the host will not prevent the borked version from being redeployed on minions
4. There is no host cache on the salt-ssh machine that might have cached this file
5. Rebuilding salt and msgpack without -march=native does not solve the issue
6. There is no msgpack installed in the minion which could possibly be responsible

Clearly there's a bug when running salt-ssh on a Gentoo host. But to further diagnose, I have to know just where salt-ssh gets the _cmsgpack.so from that it deploys on the minions. Is it being built on-the-fly during each run?? It doesn't make any sense.

Steps to Reproduce Issue

Merely try to run state.highstate on a minion with older CPU.

Versions Report

Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: 1.13.2
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: Not Installed
        libnacl: 1.6.1
       M2Crypto: 0.31.0
           Mako: 1.1.0
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.6.6
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.17 (default, Feb 26 2020, 13:54:06)
   python-gnupg: Not Installed
         PyYAML: 5.3
          PyZMQ: 17.1.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: gentoo 2.6
         locale: UTF-8
        machine: x86_64
        release: 5.4.17-ck-HH
         system: Linux
        version: Gentoo Base System 2.6

Thank you! Here's hoping I can get this fixed.

Cheers

Start by tracing salt-ssh's deployment of msgpack/_cmsgpack.so, comparing the host path /usr/lib64/python2.7/site-packages/msgpack/_cmsgpack.so with the minion path /var/tmp/.salt-*/pyall/msgpack/_cmsgpack.so using the reported strace observations. Reproduce with state.highstate on an older CPU and determine the source of the incompatible binary; done means the deployment no longer causes the invalid-opcode failure.