[BUG] TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object when trying to run ESXi proxy minion
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
Trying to run an ESXi proxy minion against a standalone ESXi 7.0.3 host. When I run salt-proxy --proxyid='hv01' -l debug, I get the following output over and over:
[CRITICAL] Unexpected error while connecting to salt01.example.com
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 79, in _get_top_file_envs
return __context__["saltutil._top_file_envs"]
File "/usr/lib/python3.6/site-packages/salt/loader/context.py", line 78, in __getitem__
return self.value()[item]
KeyError: 'saltutil._top_file_envs'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1140, in _connect_minion
yield minion.connect_master(failed=failed)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3.6/site-packages/salt/minion.py", line 1369, in connect_master
yield self._post_master_init(master)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib64/python3.6/types.py", line 182, in send
return self.__wrapped.send(val)
File "/usr/lib/python3.6/site-packages/salt/metaproxy/proxy.py", line 133, in post_master_init
self.functions["saltutil.sync_all"](saltenv=self.opts["saltenv"])
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
return callable(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 1050, in sync_all
ret["clouds"] = sync_clouds(saltenv, False, extmod_whitelist, extmod_blacklist)
File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 764, in sync_clouds
ret = _sync("clouds", saltenv, extmod_whitelist, extmod_blacklist)
File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 101, in _sync
saltenv = _get_top_file_envs()
File "/usr/lib/python3.6/site-packages/salt/modules/saltutil.py", line 85, in _get_top_file_envs
envs = list(st_.top_matches(top).keys()) or "base"
File "/usr/lib/python3.6/site-packages/salt/state.py", line 3968, in top_matches
ext_matches = self._master_tops()
File "/usr/lib/python3.6/site-packages/salt/state.py", line 3987, in _master_tops
return self.client.master_tops()
File "/usr/lib/python3.6/site-packages/salt/fileclient.py", line 1418, in master_tops
return self.channel.send(load)
File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 351, in send
load, tries=tries, timeout=timeout, raw=raw
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 318, in _crypted_transfer
ret = yield _do_transfer()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 299, in _do_transfer
self._package_load(self.auth.crypticle.dumps(load)),
File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 1519, in dumps
toencrypt = self.PICKLE_PAD + salt.payload.dumps(obj)
File "/usr/lib/python3.6/site-packages/salt/payload.py", line 168, in dumps
msg, default=ext_type_encoder, use_bin_type=use_bin_type
File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/usr/lib64/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
[Previous line repeated 1 more time]
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 281, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object
[DEBUG ] Connecting to master. Attempt 1 of 1
[DEBUG ] "salt01.example.com" Not an IP address? Assuming it is a hostname.
[DEBUG ] Master URI: tcp://[2001:db8:2:1516::11]:4506
[DEBUG ] Re-using AsyncAuth for ('/etc/salt/pki/proxy', 'hv01', 'tcp://[2001:db8:2:1516::11]:4506')
[DEBUG ] Generated random reconnect delay between '1000ms' and '11000ms' (1883)
[DEBUG ] Setting zmq_reconnect_ivl to '1883ms'
[DEBUG ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG ] Connecting the Minion to the Master publish port, using the URI: tcp://[2001:db8:2:1516::11]:4505
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/proxy/minion.pem
[DEBUG ] No metaproxy key found in opts for id hv01. Defaulting to standard proxy minion
[DEBUG ] LazyLoaded proxy.post_master_init
[DEBUG ] subclassed LazyLoaded _post_master_init
[DEBUG ] Re-using AsyncAuth for ('/etc/salt/pki/proxy', 'hv01', 'tcp://[2001:db8:2:1516::11]:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://[2001:db8:2:1516::11]:4506
[DEBUG ] Trying to connect to: tcp://[2001:db8:2:1516::11]:4506
[ERROR ] Exception getting pillar:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/pillar/__init__.py", line 242, in compile_pillar
dictkey="pillar",
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 232, in crypted_transfer_decode_dictentry
self._package_load(self.auth.crypticle.dumps(load)),
File "/usr/lib/python3.6/site-packages/salt/crypt.py", line 1519, in dumps
toencrypt = self.PICKLE_PAD + salt.payload.dumps(obj)
File "/usr/lib/python3.6/site-packages/salt/payload.py", line 168, in dumps
msg, default=ext_type_encoder, use_bin_type=use_bin_type
File "/usr/lib/python3.6/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/usr/lib64/python3.6/site-packages/msgpack/__init__.py", line 46, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 284, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 290, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 287, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 234, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 263, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 281, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'vim.host.IpConfig.IpV6Address' object
[ERROR ] Error while bringing up minion for multi-master. Is master at salt01.example.com responding?
Setup
/srv/pillar/hv01.sls on master (salt01)
proxy:
proxytype: esxi
host: hv01.example.com
username: < removed >
passwords:
- < removed >
/etc/salt/proxy.d/proxy.conf on proxy host (salt-proxy02)
master:
- salt01.example.com
- salt02.example.com
Steps to Reproduce the behavior
Start the proxy minion: salt-proxy --proxyid='hv01' -l debug
Expected behavior
Proxy minion should connect to master
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3004.1
Dependency Versions:
cffi: 1.11.5
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: 0.26.8
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: 0.26.4
Python: 3.6.8 (default, Nov 17 2021, 16:10:06)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 19.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: almalinux 8.5 Arctic Sphynx
locale: UTF-8
machine: x86_64
release: 4.18.0-348.20.1.el8_5.x86_64
system: Linux
version: AlmaLinux 8.5 Arctic Sphynx
Additional context
Both master and proxy host are running on AlmaLinux 8.5
pyVmomi version on proxy host is 7.0.3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the shown ESXi proxy configuration and inspect the paths named in the traceback, especially salt/modules/saltutil.py, salt/metaproxy/proxy.py, salt/transport/zeromq.py, and salt/payload.py. Trace which proxy data reaches msgpack serialization, then verify that the proxy connects to the master without the IpV6Address TypeError and that relevant regression coverage passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100