file.managed: list source eagerly calls cp.list_master before checking HTTP sources
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
When file.managed uses a list of sources, file.source_list calls cp.list_master and cp.list_master_dirs before iterating through the list — even if the first source is HTTP and would resolve immediately.
https://github.com/saltstack/salt/blob/master/salt/modules/file.py#L4730-L4740
mfiles = [(f, saltenv) for f in __salt__["cp.list_master"](saltenv)]
mdirs = [(d, saltenv) for d in __salt__["cp.list_master_dirs"](saltenv)]
This means every state with a list source pays the cost of a full fileserver scan upfront, regardless of source order.
Reproduce
/tmp/test_file:
file.managed:
- source:
- https://example.com/myfile
- salt://myfile
- skip_verify: True
time salt-call state.apply test
With a large fileserver (10k+ files), this is noticeably slow or times out — even though the HTTP source would succeed in milliseconds.
Expected
Salt should check sources in order and only call cp.list_master if it actually reaches a salt:// entry.
Suggested Fix
Lazy-evaluate — only fetch the file listing when a salt:// source is reached:
_cache = {}
def _get_mfiles(env):
if env not in _cache:
_cache[env] = set(__salt__["cp.list_master"](env))
return _cache[env]
Replace (path, senv) in mfiles with path in _get_mfiles(senv).
Also changes lookup from O(n) list scan to O(1) set membership.
Versions
All versions affected. Confirmed on 3006.x, code unchanged in master.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/modules/file.py around lines 4730-4740 and trace file.source_list while reproducing with the provided file.managed state and salt-call state.apply test command. Ensure HTTP sources are checked in order without an upfront fileserver listing, while salt:// sources still resolve correctly and repeated lookups avoid unnecessary scans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100