[Bug]: `state.sls(queue=True)` spools `fun=None` for in-process calls, resulting in `AttributeError` when the queue drains
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
What happened?
As far as I can tell, when a state function is called with queue=True in-process (i.e. via __salt__["state.sls"](...) from another execution/state module, rather than from a master publish) and a prior state run is active, the job is spooled to the state queue directory with its fun field set to None. When the queue later drains and the minion tries to dispatch the spooled payload, data["fun"] is None,the call to None.split(".") raises an AttributeError.
Root cause
(This was determined by Claude Code and I checked it as best as I could, but I'm not familiar with the Salt codebase so this may not be 100% correct).
1. The spooled payload takes fun from the __pub_fun kwarg.
In _check_queue, when a conflict is detected and the job is queued, the persisted payload is built like this:
_check_queue is called by every queue-aware state function (sls, highstate, top, apply_, etc.) — e.g. https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/modules/state.py#L1510
2. __pub_* kwargs seemingly exist only when the function was invoked from a master publish — not for in-process calls.
kwargs are created in load_args_and_kwargs, which packs the publish data dict into the call kwargs, prefixing each key with __pub_:
Here data is the publish job payload ({"fun": ..., "arg": ..., "jid": ..., "tgt": ..., "user": ..., "ret": ...}).
load_args_and_kwargs is called only inside the minion's job-execution thread, which passes that publish payload as data:
This path thus runs only when the minion executes a job that was dispatched from a publish (_thread_return / _execute_job_function), and an in-process call (__salt__["state.sls"](...)) does not go through that path.
Therefore, for an in-process caller, kwargs contains no __pub_* keys, so kwargs.get("__pub_fun") returns None, and the payload is spooled with "fun": None. (Every other __pub_*-derived field — tgt, arg, ret, user — is similarly absent, but fun=None is the one that later crashes.)
**3. On drain, the payload is dispatched unmodified, with fun set to None
The state-queue drain loads the persisted payload, repairs the JID from the filename, sets a bypass flag, and dispatches — but never touches data["fun"]:
4. fun=None slips through the dispatch guards and crashes.
-
The string-only guard is skipped for
None:https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/minion.py#L2262
-
The multi-func fork treats
Noneas not a list/tuple, so it goes to the single-function path: -
In
_thread_return,function_namebecomesNone:https://github.com/saltstack/salt/blob/c3e5bb1981a025a69bf94269282838ab618079c8/salt/minion.py#L2892
-
Noneis not infunctions, so the missing-function path runs, callingmissing_fun_string(None)and thenfunction_name.split("."):missing_fun_stringitself dereferencesNonefirst, so it actually raises here:
Either way the drained job raises AttributeError: 'NoneType' object has no attribute 'split' and never executes.
Impact
Any in-process caller that does state.sls(..., queue=True) (or highstate/top/apply with queue=True) while another state run is active will have its stage silently spooled and then dropped with an AttributeError when the queue drains. Master-published queue=True jobs are unaffected because they carry __pub_fun.
Steps to reproduce
- Start a long-running highstate on a minion.
- From within an execution/state module (or any in-process context, e.g. an orchestrating custom module), call
__salt__["state.sls"]("some_state", queue=True)while that highstate is still running. - The call returns
{"result": True, "queued": True, "__no_return__": True, ...}(maybe that's a bug too? Not sure if a queued job should claim that the result is True) - When the highstate finishes and the state queue drains, the minion log shows
AttributeError: 'NoneType' object has no attribute 'split', andsome_statenever runs.
Type of salt install
Official deb
Major version
3006.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
ubuntu-24.04
salt --versions-report output
Salt Version:
Salt: 3006.23+3.gf52b9c0
Python Version:
Python: 3.10.19 (main, Feb 17 2026, 23:32:03) [GCC 11.2.0]
Dependency Versions:
cffi: 2.0.0
cherrypy: 18.10.0
cryptography: 46.0.5
dateutil: 2.8.1
docker-py: 7.2.0
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: 1.6.4
looseversion: 1.0.2
M2Crypto: 0.39.0
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 24.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.12.2
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.22.4
smmap: Not Installed
timelib: 0.3.0
Tornado: 4.5.3
ZMQ: 4.3.4
Salt Extensions:
saltext.prometheus: 2.2.0
saltext.vault: 1.5.0
System Versions:
dist: ubuntu 24.04.2 noble
locale: utf-8
machine: x86_64
release: 6.11.0-19-generic
system: Linux
version: Ubuntu 24.04.2 noble
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/modules/state.py at _check_queue and the queue-aware state functions, then trace the payload through salt/minion.py's job-loading and queue-drain paths. Compare in-process and master-published calls, including salt/loader/lazy.py's missing-function handling. Reproduce the queued in-process call and verify that the drained state runs without the AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100