saltstack / saltstack/salt

[BUG] TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object when trying to run ESXi proxy minion

Open
#61,982 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Proxy-Minion severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Trying to run an ESXi proxy minion against a standalone ESXi 7.0.3 host. When I run salt-proxy --proxyid='hv01' -l debug, I get the following output over and over:

[CRITICAL] Unexpected error while connecting to salt01.example.com
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 79, in _get_top_file_envs
    return __context__["saltutil._top_file_envs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/context.py", line 78, in __getitem__
    return self.value()[item]
KeyError: 'saltutil._top_file_envs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1140, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1369, in connect_master
    yield self._post_master_init(master)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib64/python3.6/types.py", line 182, in send
    return self.__wrapped.send(val)
  File "/usr/lib/python3.6/site-packages/salt/metaproxy/proxy.py", line 133, in post_master_init
    self.functions["saltutil.sync_all"](saltenv=self.opts["saltenv"])
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1050, in sync_all
    ret["clouds"] = sync_clouds(saltenv, False, extmod_whitelist, extmod_blacklist)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 764, in sync_clouds
    ret = _sync("clouds", saltenv, extmod_whitelist, extmod_blacklist)
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 101, in _sync
    saltenv = _get_top_file_envs()
  File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 85, in _get_top_file_envs
    envs = list(st_.top_matches(top).keys()) or "base"
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 3968, in top_matches
    ext_matches = self._master_tops()
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 3987, in _master_tops
    return self.client.master_tops()
  File "/usr/lib/python3.6/site-packages/salt/fileclient.py", line 1418, in master_tops
    return self.channel.send(load)
  File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 351, in send
    load, tries=tries, timeout=timeout, raw=raw
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 318, in _crypted_transfer
    ret = yield _do_transfer()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 299, in _do_transfer
    self._package_load(self.auth.crypticle.dumps(load)),
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 1519, in dumps
    toencrypt = self.PICKLE_PAD + salt.payload.dumps(obj)
  File "/usr/lib/python3.6/site-packages/salt/payload.py", line 168, in dumps
    msg, default=ext_type_encoder, use_bin_type=use_bin_type
  File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
    return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
  File "/usr/lib64/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  [Previous line repeated 1 more time]
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 281, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object
[DEBUG   ] Connecting to master. Attempt 1 of 1
[DEBUG   ] "salt01.example.com" Not an IP address? Assuming it is a hostname.
[DEBUG   ] Master URI: tcp://[2001:db8:2:1516::11]:4506
[DEBUG   ] Re-using AsyncAuth for ('/etc/salt/pki/proxy', 'hv01', 'tcp://[2001:db8:2:1516::11]:4506')
[DEBUG   ] Generated random reconnect delay between '1000ms' and '11000ms' (1883)
[DEBUG   ] Setting zmq_reconnect_ivl to '1883ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG   ] Connecting the Minion to the Master publish port, using the URI: tcp://[2001:db8:2:1516::11]:4505
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] Loaded minion key: /etc/salt/pki/proxy/minion.pem
[DEBUG   ] No metaproxy key found in opts for id hv01. Defaulting to standard proxy minion
[DEBUG   ] LazyLoaded proxy.post_master_init
[DEBUG   ] subclassed LazyLoaded _post_master_init
[DEBUG   ] Re-using AsyncAuth for ('/etc/salt/pki/proxy', 'hv01', 'tcp://[2001:db8:2:1516::11]:4506')
[DEBUG   ] Connecting the Minion to the Master URI (for the return server): tcp://[2001:db8:2:1516::11]:4506
[DEBUG   ] Trying to connect to: tcp://[2001:db8:2:1516::11]:4506
[ERROR   ] Exception getting pillar:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 242, in compile_pillar
    dictkey="pillar",
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 232, in crypted_transfer_decode_dictentry
    self._package_load(self.auth.crypticle.dumps(load)),
  File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 1519, in dumps
    toencrypt = self.PICKLE_PAD + salt.payload.dumps(obj)
  File "/usr/lib/python3.6/site-packages/salt/payload.py", line 168, in dumps
    msg, default=ext_type_encoder, use_bin_type=use_bin_type
  File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
    return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
  File "/usr/lib64/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 281, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object
[ERROR   ] Error while bringing up minion for multi-master. Is master at salt01.example.com responding?

Setup
/srv/pillar/hv01.sls on master (salt01)

proxy:
  proxytype: esxi
  host: hv01.example.com
  username: < removed >
  passwords: 
    - < removed >

/etc/salt/proxy.d/proxy.conf on proxy host (salt-proxy02)

master:
  - salt01.example.com
  - salt02.example.com

Steps to Reproduce the behavior
Start the proxy minion: salt-proxy --proxyid='hv01' -l debug

Expected behavior
Proxy minion should connect to master

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.11.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: 0.26.8
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: 0.26.4
        Python: 3.6.8 (default, Nov 17 2021, 16:10:06)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: almalinux 8.5 Arctic Sphynx
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-348.20.1.el8_5.x86_64
        system: Linux
       version: AlmaLinux 8.5 Arctic Sphynx

Additional context
Both master and proxy host are running on AlmaLinux 8.5

pyVmomi version on proxy host is 7.0.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the shown ESXi proxy configuration and inspect the paths named in the traceback, especially salt/modules/saltutil.py, salt/metaproxy/proxy.py, salt/transport/zeromq.py, and salt/payload.py. Trace which proxy data reaches msgpack serialization, then verify that the proxy connects to the master without the IpV6Address TypeError and that relevant regression coverage passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.