saltstack / saltstack/salt

[BUG] Occasional error reporting:FAILED transport/test_ipc.py::IPCMessagePubSubCase::test_async_reading_streamclosederror - RuntimeError: IOLoop is already running

Open
#68,266 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Occasional error reporting:FAILED transport/test_ipc.py::IPCMessagePubSubCase::test_async_reading_streamclosederror - RuntimeError: IOLoop is already running

self = <tests.unit.transport.test_ipc.IPCMessagePubSubCase testMethod=test_async_reading_streamclosederror>

    @salt.ext.tornado.testing.gen_test
    def test_async_reading_streamclosederror(self):
        client1 = self.sub_channel
        call_cnt = []

        # Create a watchdog to be safe from hanging in sync loops (what old code did)
        evt = threading.Event()

        def close_server():
            if evt.wait(0.001):
                return
            client1.close()
            self.stop()

        watchdog = threading.Thread(target=close_server)
        watchdog.start()

        # Runs in ioloop thread so we're safe from race conditions here
        def handler(raw):
            pass

        try:
            ret1 = yield client1.read_async(handler)
>           self.wait()

/home/stage/root/spack-stage-py-salt-3006.8-ly6kzp7i3mswk5umrb4qmtykof3l6qz7/spack-src/tests/unit/transport/test_ipc.py:151:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/stage/root/spack-stage-py-salt-3006.8-ly6kzp7i3mswk5umrb4qmtykof3l6qz7/spack-src/salt/ext/tornado/testing.py:328: in wait
    self.io_loop.start()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <salt.ext.tornado.platform.epoll.EPollIOLoop object at 0xffff8ea5b9a0>

    def start(self):
        if self._running:
>           raise RuntimeError("IOLoop is already running")
E           RuntimeError: IOLoop is already running

/home/stage/root/spack-stage-py-salt-3006.8-ly6kzp7i3mswk5umrb4qmtykof3l6qz7/spack-src/salt/ext/tornado/ioloop.py:757: RuntimeError

FAILED transport/test_ipc.py::IPCMessagePubSubCase::test_async_reading_streamclosederror - RuntimeError: IOLoop is already running

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • [Yes ] on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
1、spack install --keep-stage --dont-restage --no-checksum -y py-salt@3006.8
2、spack install --overwrite --keep-stage --no-checksum -y py-pip@20.2
3、pip install --use-feature=2020-resolver looseversion Jinja2 contextvars jmespath msgpack==1.0.0 psutil==6.1.1 pycryptodomex==3.9.8 pyzmq==25.0.2 more_itertools looseversion attr pytest==7.4.0 boto3 boto pytest-helpers-namespace
4、pip install -r /home/stage/root/spack-stage-py-salt-3006.8-ly6kzp7i3mswk5umrb4qmtykof3l6qz7/spack-src/requirements/pytest.txt
5、cd /home/stage/root/spack-stage-py-salt-3006.8-ly6kzp7i3mswk5umrb4qmtykof3l6qz7/spack-src/tests/unit
6、pytest -s -v

Expected behavior
no errors

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt --version
salt 3006.8 (Sulfur)

Environmental Information:
cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"

PASTE HERE

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/unit/transport/test_ipc.py, especially IPCMessagePubSubCase.test_async_reading_streamclosederror and its call to self.wait(). Read salt/ext/tornado/testing.py and salt/ext/tornado/ioloop.py around wait() and start() to understand the overlapping loop state. Done means the test no longer intermittently raises RuntimeError: IOLoop is already running when the unit suite is run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.