saltstack / saltstack/salt

file.managed: list source eagerly calls cp.list_master before checking HTTP sources

Open
#69,804 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

When file.managed uses a list of sources, file.source_list calls cp.list_master and cp.list_master_dirs before iterating through the list — even if the first source is HTTP and would resolve immediately.

https://github.com/saltstack/salt/blob/master/salt/modules/file.py#L4730-L4740

mfiles = [(f, saltenv) for f in __salt__["cp.list_master"](saltenv)]
mdirs = [(d, saltenv) for d in __salt__["cp.list_master_dirs"](saltenv)]

This means every state with a list source pays the cost of a full fileserver scan upfront, regardless of source order.

Reproduce

/tmp/test_file:
  file.managed:
    - source:
      - https://example.com/myfile
      - salt://myfile
    - skip_verify: True
time salt-call state.apply test

With a large fileserver (10k+ files), this is noticeably slow or times out — even though the HTTP source would succeed in milliseconds.

Expected

Salt should check sources in order and only call cp.list_master if it actually reaches a salt:// entry.

Suggested Fix

Lazy-evaluate — only fetch the file listing when a salt:// source is reached:

_cache = {}

def _get_mfiles(env):
    if env not in _cache:
        _cache[env] = set(__salt__["cp.list_master"](env))
    return _cache[env]

Replace (path, senv) in mfiles with path in _get_mfiles(senv).

Also changes lookup from O(n) list scan to O(1) set membership.

Versions

All versions affected. Confirmed on 3006.x, code unchanged in master.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/modules/file.py around lines 4730-4740 and trace file.source_list while reproducing with the provided file.managed state and salt-call state.apply test command. Ensure HTTP sources are checked in order without an upfront fileserver listing, while salt:// sources still resolve correctly and repeated lookups avoid unnecessary scans.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.