saltstack / saltstack/salt

[BUG] etcd cache: ls() returns nested leaf key names instead of a bank's immediate children, breaking grain (-G) targeting

Open
#69,616 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description

With cache: etcd (the etcd v2 minion data cache), grain targeting from the master matches no minions:

# salt -G 'os:*' test.ping
No minions matched the target. No command was sent, no jid was assigned.

Glob targeting (salt '*' test.ping) works fine, so the minions are connected and their data is cached.

The root cause is salt.cache.etcd_cache.ls() (through its _walk() helper): it recurses through the whole bank subtree and returns the flattened leaf key names, instead of the bank's immediate children. The master stores each minion's data under a per-minion sub-bank, minions/<minion_id>, with data and mine keys inside it. So:

cache.list("minions")   # etcd    -> ['data', 'mine', 'data', 'mine', ...]
cache.list("minions")   # localfs -> ['minion1', 'minion2', ...]   (correct)

salt/utils/minions.py iterates cache.list("minions") and then does cache.fetch("minions/<id>", "data") for each entry. Because the list contains data/mine instead of minion IDs, fetch("minions/data", "data") resolves to nothing and no minion is ever evaluated, so grain, pillar, range, and the cache portion of compound matching all return an empty set.

Every other cache backend (localfs, redis, consul, mysql) returns a bank's immediate children from list(). etcd is the outlier.

Setup

  • Salt master with cache: etcd in the master config
  • minion_data_cache: True (the default)
  • python-etcd installed on the master; etcd reachable (etcd v2, or etcd v3 with the v2 API enabled)
  • container (podman) for the reproduction below; also observed on a normal 3006.x master

Steps to Reproduce the behavior

Minimal, self-contained reproduction against a real etcd v2 server. It seeds the cache exactly as the master's _pillar handler does and then runs the real CkMinions grain matcher, for both the etcd and localfs backends:

import salt.config, salt.cache, salt.utils.minions

def run(backend):
    opts = salt.config.master_config("/etc/salt/master")   # cache: <backend>, minion_data_cache: True
    cache = salt.cache.factory(opts)
    cache.store("minions/minion1", "data",
                {"grains": {"id": "minion1", "os": "Debian"}, "pillar": {}})
    ckm = salt.utils.minions.CkMinions(opts)
    print(backend, "list:", cache.list("minions"),
          "-G os:Debian ->", ckm.check_minions("os:Debian", "grain")["minions"])

run("etcd")      # etcd    list: ['data'] -G os:Debian -> []
run("localfs")   # localfs list: ['minion1'] -G os:Debian -> ['minion1']

Output:

etcd     list: ['data']     -G os:Debian -> []
localfs  list: ['minion1']  -G os:Debian -> ['minion1']

cache.fetch("minions/minion1", "data") returns the correct data on both backends, so the defect is isolated to list().

Expected behavior

cache.list("minions") returns the cached minion IDs, and -G grain targeting matches the same minions it would with cache: localfs.

Versions Report

salt --versions-report (reproduction environment)
Salt Version:
    Salt: 3006.26

Dependency Versions:
    python-etcd: 0.4.5

System Versions:
    Python: 3.10
    etcd server: 2.x (v2 API)

Reproduced on 3006.x and 3007.x.

Additional context

  • The symptom (-G returns nothing) is on 3006.x and 3007.x, where the master enumerates the nested minions bank. On 3008.x/master the minion-data cache was refactored to a flat grains bank, which masks this particular symptom, but salt/cache/etcd_cache.py is byte-identical on all four branches and the same ls() contract violation still affects other nested-bank callers (for example the cache.migrate runner and thorium).
  • Prior art: #55305 is the same root cause (closed as stale; the reporter proposed the same fix). #56171 is a matching mine.get grain-targeting symptom. #57377 / #69235 fixed only the empty-folder infinite-recursion crash in _walk, not this contract bug. #66102 is the same class of bug in the mysql cache driver.
  • Fix + tests in the linked PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/cache/etcd_cache.py at ls() and its _walk() helper, then inspect salt/utils/minions.py where cache.list("minions") feeds grain matching. Reproduce with the provided etcd and localfs example; done means etcd lists immediate minion children and CkMinions grain targeting matches the localfs result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.