[BUG] salt.loaded.int.grains.metadata_gce.__virtual__() fails with stack trace in logs on AWS
Open
Nobody has claimed this yet.
bug
needs-triage
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
The Salt minion emits stack traces into the logs.
2025-01-09 11:11:45,881 [salt.loader.lazy :1191][ERROR ][21166] Exception raised when processing __virtual__ function for salt.loaded.int.grains.metadata_gce. Module will not be loaded: Cannot run the event loop while another loop is running
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1174, in _process_virtual
virtual = self.run(virtual_attr)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata_gce.py", line 32, in __virtual__
googletest = http.query(HOST, status=True, headers=True)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/http.py", line 633, in query
HTTPClient(max_body_size=max_body)
File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/httpclient.py", line 109, in __init__
self._async_client = self._io_loop.run_sync(make_client)
File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 521, in run_sync
self.start()
File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 195, in start
self.asyncio_loop.run_forever()
File "/opt/saltstack/salt/lib/python3.10/asyncio/base_events.py", line 592, in run_forever
self._check_running()
File "/opt/saltstack/salt/lib/python3.10/asyncio/base_events.py", line 586, in _check_running
raise RuntimeError(
RuntimeError: Cannot run the event loop while another loop is running
2025-01-09 11:11:45,882 [salt.loader.lazy :1203][WARNING ][21166] salt.loaded.int.grains.metadata_gce.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'metadata_gce', please fix this.
Setup
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service - AWS EC2 eu-west-1 r6i.xlarge
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
Start Salt minion.
Expected behavior
No stack traces in the logs.
Versions Report
salt --versions-report
Master:
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: amzn 2023.6.20241121
locale: utf-8
machine: x86_64
release: 6.1.115-126.197.amzn2023.x86_64
system: Linux
version: Amazon Linux 2023.6.20241121
Minions:
Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: 18.8.0
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: centos 7.9.2009 Core
locale: utf-8
machine: x86_64
release: 3.10.0-1160.80.1.el7.x86_64
system: Linux
version: CentOS Linux 7.9.2009 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with salt/grains/metadata_gce.py at virtual, then trace the HTTP call through salt/utils/http.py and the Tornado event-loop path shown in the traceback. Reproduce by starting a Salt minion in the reported AWS onedir setup, and verify that the metadata grain does not emit stack traces or the wrongly-returning warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100