saltstack / saltstack/salt

[BUG] salt.loaded.int.grains.metadata_gce.__virtual__() fails with stack trace in logs on AWS

Open
#67,132 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

The Salt minion emits stack traces into the logs.

2025-01-09 11:11:45,881 [salt.loader.lazy :1191][ERROR   ][21166] Exception raised when processing __virtual__ function for salt.loaded.int.grains.metadata_gce. Module will not be loaded: Cannot run the event loop while another loop is running
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1174, in _process_virtual
    virtual = self.run(virtual_attr)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata_gce.py", line 32, in __virtual__
    googletest = http.query(HOST, status=True, headers=True)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/http.py", line 633, in query
    HTTPClient(max_body_size=max_body)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/httpclient.py", line 109, in __init__
    self._async_client = self._io_loop.run_sync(make_client)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/ioloop.py", line 521, in run_sync
    self.start()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 195, in start
    self.asyncio_loop.run_forever()
  File "/opt/saltstack/salt/lib/python3.10/asyncio/base_events.py", line 592, in run_forever
    self._check_running()
  File "/opt/saltstack/salt/lib/python3.10/asyncio/base_events.py", line 586, in _check_running
    raise RuntimeError(
RuntimeError: Cannot run the event loop while another loop is running
2025-01-09 11:11:45,882 [salt.loader.lazy :1203][WARNING ][21166] salt.loaded.int.grains.metadata_gce.__virtual__() is wrongly returning `None`. It should either return `True`, `False` or a new name. If you're the developer of the module 'metadata_gce', please fix this.

Setup

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service - AWS EC2 eu-west-1 r6i.xlarge
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior

Start Salt minion.

Expected behavior

No stack traces in the logs.

Versions Report

salt --versions-report

Master:

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: amzn 2023.6.20241121
        locale: utf-8
       machine: x86_64
       release: 6.1.115-126.197.amzn2023.x86_64
        system: Linux
       version: Amazon Linux 2023.6.20241121

Minions:

Salt Version:
          Salt: 3007.1

Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: centos 7.9.2009 Core
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.80.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7.9.2009 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/grains/metadata_gce.py at virtual, then trace the HTTP call through salt/utils/http.py and the Tornado event-loop path shown in the traceback. Reproduce by starting a Salt minion in the reported AWS onedir setup, and verify that the metadata grain does not emit stack traces or the wrongly-returning warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.