saltstack / saltstack/salt

[Bug]: `state.sls(queue=True)` spools `fun=None` for in-process calls, resulting in `AttributeError` when the queue drains

Open
#69,909 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

What happened?

As far as I can tell, when a state function is called with queue=True in-process (i.e. via __salt__["state.sls"](...) from another execution/state module, rather than from a master publish) and a prior state run is active, the job is spooled to the state queue directory with its fun field set to None. When the queue later drains and the minion tries to dispatch the spooled payload, data["fun"] is None,the call to None.split(".") raises an AttributeError.

Root cause

(This was determined by Claude Code and I checked it as best as I could, but I'm not familiar with the Salt codebase so this may not be 100% correct).

1. The spooled payload takes fun from the __pub_fun kwarg.

In _check_queue, when a conflict is detected and the job is queued, the persisted payload is built like this:

https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/modules/state.py#L523-L531

_check_queue is called by every queue-aware state function (sls, highstate, top, apply_, etc.) — e.g. https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/modules/state.py#L1510

2. __pub_* kwargs seemingly exist only when the function was invoked from a master publish — not for in-process calls.

kwargs are created in load_args_and_kwargs, which packs the publish data dict into the call kwargs, prefixing each key with __pub_:

https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/minion.py#L471-L474

Here data is the publish job payload ({"fun": ..., "arg": ..., "jid": ..., "tgt": ..., "user": ..., "ret": ...}).

load_args_and_kwargs is called only inside the minion's job-execution thread, which passes that publish payload as data:

https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/minion.py#L2814-L2820

This path thus runs only when the minion executes a job that was dispatched from a publish (_thread_return / _execute_job_function), and an in-process call (__salt__["state.sls"](...)) does not go through that path.

Therefore, for an in-process caller, kwargs contains no __pub_* keys, so kwargs.get("__pub_fun") returns None, and the payload is spooled with "fun": None. (Every other __pub_*-derived field — tgt, arg, ret, user — is similarly absent, but fun=None is the one that later crashes.)

**3. On drain, the payload is dispatched unmodified, with fun set to None

The state-queue drain loads the persisted payload, repairs the JID from the filename, sets a bypass flag, and dispatches — but never touches data["fun"]:

https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/minion.py#L4793-L4846

4. fun=None slips through the dispatch guards and crashes.

Either way the drained job raises AttributeError: 'NoneType' object has no attribute 'split' and never executes.

Impact

Any in-process caller that does state.sls(..., queue=True) (or highstate/top/apply with queue=True) while another state run is active will have its stage silently spooled and then dropped with an AttributeError when the queue drains. Master-published queue=True jobs are unaffected because they carry __pub_fun.

Steps to reproduce
  1. Start a long-running highstate on a minion.
  2. From within an execution/state module (or any in-process context, e.g. an orchestrating custom module), call __salt__["state.sls"]("some_state", queue=True) while that highstate is still running.
  3. The call returns {"result": True, "queued": True, "__no_return__": True, ...} (maybe that's a bug too? Not sure if a queued job should claim that the result is True)
  4. When the highstate finishes and the state queue drains, the minion log shows AttributeError: 'NoneType' object has no attribute 'split', and some_state never runs.
Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

ubuntu-24.04

salt --versions-report output
Salt Version:
               Salt: 3006.23+3.gf52b9c0

Python Version:
             Python: 3.10.19 (main, Feb 17 2026, 23:32:03) [GCC 11.2.0]

Dependency Versions:
               cffi: 2.0.0
           cherrypy: 18.10.0
       cryptography: 46.0.5
           dateutil: 2.8.1
          docker-py: 7.2.0
              gitdb: Not Installed
          gitpython: Not Installed
             Jinja2: 3.1.6
            libgit2: 1.6.4
       looseversion: 1.0.2
           M2Crypto: 0.39.0
               Mako: Not Installed
            msgpack: 1.0.2
       msgpack-pure: Not Installed
       mysql-python: Not Installed
          packaging: 24.0
          pycparser: 2.21
           pycrypto: Not Installed
       pycryptodome: 3.19.1
             pygit2: 1.12.2
       python-gnupg: 0.4.8
             PyYAML: 6.0.1
              PyZMQ: 23.2.0
             relenv: 0.22.4
              smmap: Not Installed
            timelib: 0.3.0
            Tornado: 4.5.3
                ZMQ: 4.3.4

Salt Extensions:
 saltext.prometheus: 2.2.0
      saltext.vault: 1.5.0

System Versions:
               dist: ubuntu 24.04.2 noble
             locale: utf-8
            machine: x86_64
            release: 6.11.0-19-generic
             system: Linux
            version: Ubuntu 24.04.2 noble

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/modules/state.py at _check_queue and the queue-aware state functions, then trace the payload through salt/minion.py's job-loading and queue-drain paths. Compare in-process and master-published calls, including salt/loader/lazy.py's missing-function handling. Reproduce the queued in-process call and verify that the drained state runs without the AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.