saltstack / saltstack/salt

[Bug]: proxy minions log "Error during asyncio shutdown: The future belongs to a different loop" on Python 3.14

Open
#70,226 0 comments 0 reactions 0 assignees View on GitHub

A pull request for this has already been merged.

  • #70227 by @ggiesen — merged
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

On Python 3.14 every proxy minion logs a wall of

[ERROR   ] Error during asyncio shutdown: The future belongs to a different loop than the one specified as the loop argument

during startup. On a freshly started single proxy that is the entire log: 20 lines, all of them this. A deltaproxy with two sub-proxies produces 58.

The message says "shutdown" but these appear while the proxy is coming up, which makes it look alarming and unrelated to what is actually happening.

Root cause

salt/utils/asynchronous.py, in close():

pending_tasks = [
    task for task in asyncio.all_tasks(self.asyncio_loop) if not task.done()
]
if pending_tasks:
    for task in pending_tasks:
        task.cancel()
    gathered = asyncio.gather(*pending_tasks, return_exceptions=True)
    try:
        self.asyncio_loop.run_until_complete(gathered)

asyncio.gather has taken no loop argument since 3.10, so it resolves the loop from the calling context. close() runs outside the loop it is tearing down, and the tasks belong to self.asyncio_loop. On Python 3.14 ensure_future rejects that mismatch; earlier versions took the loop from the first future and let it through.

Instrumenting the call site confirms the loops never match:

asyncio_loop=138915023434640 current=138915031292560 same=False ntasks=1
asyncio_loop=138914893251600 current=138915031292560 same=False ntasks=1
...

The exception is caught by the broad except below and logged, so nothing crashes -- but the pending tasks are never drained, which is the work close() was doing.

Setup

Python 3.14.7, salt 3008.2, Linux. Reproduces with a single proxy minion; a deltaproxy just produces more of them.

Steps to reproduce

Start any proxy minion under Python 3.14 and read its log.

Observed (single proxy, -l error, whole log):

[ERROR   ] Error during asyncio shutdown: The future belongs to a different loop than the one specified as the loop argument
... 20 identical lines ...

Expected: no such errors, and the pending tasks actually drained.

Scoped on the same machine, so it is specific to proxies rather than salt generally:

occurrences
salt-call --local 0
salt-master 0
single salt-proxy 20
deltaproxy, 2 sub-proxies 58
any of the above on Python 3.10 0
Versions Report
Salt Version:
           Salt: 3008.2
Python Version:
         Python: 3.14.7

Reproduces against the current 3008.x branch head.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/utils/asynchronous.py, focusing on close() and its asyncio shutdown handling. Reproduce the startup error with a proxy minion under Python 3.14, then verify the shutdown path against the loop that owns the pending tasks. Done means proxy startup emits no asyncio shutdown errors and the pending tasks are drained.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.