saltstack / saltstack/salt

[BUG] strange grains targeting behavior after minion reboot

Open
#59,156 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Grains severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
After a minion reboots, at least some of its grains are incompletely available for targeting until running saltutil.refresh_grains against it. This is a new behavior, but I'm not sure when it appeared -- possibly with 3002.

Setup
Set up, for example, a master and minions that use the tier grain.

Steps to Reproduce the behavior

$ salt -C 'wr* and G@tier:stage' test.ping
wr-stage-003:
    True
wr-stage-002:
    True
wr-stage-005:
    True
$ salt -C 'wr-stage-002' system.reboot
wr-stage-002:

Wait a minute for the minion to reboot, and confirm:

$ salt -C 'wr-stage-002' test.ping
wr-stage-002:
    True

Here's the strange part -- this works, as expected:

$ salt -C 'wr* and G@tier:stage' test.ping
wr-stage-003:
    True
wr-stage-005:
    True
wr-stage-002:
    True

but this doesn't:

$ salt -C 'wr*2 and G@tier:stage' test.ping
No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received

even though the same command gets a correct result without the grain targeting:

$ salt -C 'wr*2' test.ping
wr-stage-002:
    True
wr-prod-012:
    True

After running

$ salt -C 'wr-stage-002' saltutil.refresh_grains
wr-stage-002:
    True

this works as expected:

$ salt -C 'wr*2 and G@tier:stage' test.ping
wr-stage-002:
    True

Expected behavior
After rebooting a minion, it should be targetable using grains.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.5
     gitpython: 2.1.11
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.5.6
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.1.2
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1

System Versions:
          dist: debian 10 buster
        locale: UTF-8
       machine: x86_64
       release: 4.19.0-13-amd64
        system: Linux
       version: Debian GNU/Linux 10 buster

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the post-reboot targeting commands with the tier grain, then compare behavior before and after saltutil.refresh_grains. Trace the minion grain refresh and compound-targeting entry points involved in those commands. Done means a rebooted minion is immediately selectable with wr*2 and G@tier:stage without manually refreshing grains.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.