[BUG] etcd cache: ls() returns nested leaf key names instead of a bank's immediate children, breaking grain (-G) targeting
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
With cache: etcd (the etcd v2 minion data cache), grain targeting from the master matches no minions:
# salt -G 'os:*' test.ping
No minions matched the target. No command was sent, no jid was assigned.
Glob targeting (salt '*' test.ping) works fine, so the minions are connected and their data is cached.
The root cause is salt.cache.etcd_cache.ls() (through its _walk() helper): it recurses through the whole bank subtree and returns the flattened leaf key names, instead of the bank's immediate children. The master stores each minion's data under a per-minion sub-bank, minions/<minion_id>, with data and mine keys inside it. So:
cache.list("minions") # etcd -> ['data', 'mine', 'data', 'mine', ...]
cache.list("minions") # localfs -> ['minion1', 'minion2', ...] (correct)
salt/utils/minions.py iterates cache.list("minions") and then does cache.fetch("minions/<id>", "data") for each entry. Because the list contains data/mine instead of minion IDs, fetch("minions/data", "data") resolves to nothing and no minion is ever evaluated, so grain, pillar, range, and the cache portion of compound matching all return an empty set.
Every other cache backend (localfs, redis, consul, mysql) returns a bank's immediate children from list(). etcd is the outlier.
Setup
- Salt master with
cache: etcdin the master config minion_data_cache: True(the default)- python-etcd installed on the master; etcd reachable (etcd v2, or etcd v3 with the v2 API enabled)
- container (podman) for the reproduction below; also observed on a normal 3006.x master
Steps to Reproduce the behavior
Minimal, self-contained reproduction against a real etcd v2 server. It seeds the cache exactly as the master's _pillar handler does and then runs the real CkMinions grain matcher, for both the etcd and localfs backends:
import salt.config, salt.cache, salt.utils.minions
def run(backend):
opts = salt.config.master_config("/etc/salt/master") # cache: <backend>, minion_data_cache: True
cache = salt.cache.factory(opts)
cache.store("minions/minion1", "data",
{"grains": {"id": "minion1", "os": "Debian"}, "pillar": {}})
ckm = salt.utils.minions.CkMinions(opts)
print(backend, "list:", cache.list("minions"),
"-G os:Debian ->", ckm.check_minions("os:Debian", "grain")["minions"])
run("etcd") # etcd list: ['data'] -G os:Debian -> []
run("localfs") # localfs list: ['minion1'] -G os:Debian -> ['minion1']
Output:
etcd list: ['data'] -G os:Debian -> []
localfs list: ['minion1'] -G os:Debian -> ['minion1']
cache.fetch("minions/minion1", "data") returns the correct data on both backends, so the defect is isolated to list().
Expected behavior
cache.list("minions") returns the cached minion IDs, and -G grain targeting matches the same minions it would with cache: localfs.
Versions Report
salt --versions-report (reproduction environment)
Salt Version:
Salt: 3006.26
Dependency Versions:
python-etcd: 0.4.5
System Versions:
Python: 3.10
etcd server: 2.x (v2 API)
Reproduced on 3006.x and 3007.x.
Additional context
- The symptom (
-Greturns nothing) is on 3006.x and 3007.x, where the master enumerates the nestedminionsbank. On 3008.x/master the minion-data cache was refactored to a flatgrainsbank, which masks this particular symptom, butsalt/cache/etcd_cache.pyis byte-identical on all four branches and the samels()contract violation still affects other nested-bank callers (for example thecache.migraterunner andthorium). - Prior art: #55305 is the same root cause (closed as stale; the reporter proposed the same fix). #56171 is a matching
mine.getgrain-targeting symptom. #57377 / #69235 fixed only the empty-folder infinite-recursion crash in_walk, not this contract bug. #66102 is the same class of bug in the mysql cache driver. - Fix + tests in the linked PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/cache/etcd_cache.py at ls() and its _walk() helper, then inspect salt/utils/minions.py where cache.list("minions") feeds grain matching. Reproduce with the provided etcd and localfs example; done means etcd lists immediate minion children and CkMinions grain targeting matches the localfs result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100