saltstack / saltstack/salt

[BUG] Minion sends duplicate /start event (3006.1, 3006.4)

Open
#66,341 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
at least seen in Windows minion 3006.1 and 3006.4, and 3006.7, if the minion is set with a multimaster list with N masters, it sends duplicate N /start event when it starts, as per the numbers of masters in the list to the same master

For instance, the minion is set with two masters, both masters receives two /start events, which cause anomalies like reactor being triggered for each event

# MINION 3006.1 / 3006.4 (two start events in the minion log)
[DEBUG   ] SaltEvent PUB socket URI: 4510
[DEBUG   ] SaltEvent PULL socket URI: 4511
[DEBUG   ] Sending event: tag = __master_req_channel_payload; data = {'id': 'minion-win-1', 'cmd': '_minion_event', 'pretag': None, 'tok': b'\x19\x94rQG\xa1\xe2"\xc3(z\x90\xa6yV\xcfE\xf5+\x8e\x9e\xb1T\xf5N8+\xd5\xc9&d\xf8\xc0\x85\x8e*$.:L\xd2G\xc3U\n\xea\xd8\xad\xf2}S\xa9]]\xb5\x86\xebCM\xae\x86\xd40\xec\xbaX\x16\x812!\xe3>\xec\xcd\xceq\xd4\x95\xb4\xfbG!\x8f-\xa8]9#\xb7\x08\xcac\xae\xd7\xe6*.\xf5\xca\xa0\xd3\xeaO\xb4\x19\x1a7\xcc DB\x8d\xeaN\x00\x8b\x98C\xccc\xa3\xbc\xf4>\xe0\xad A@\x1fp\xab7\xd8\xfb\xc2\xd1\x1c\xbf\x9a\x03K\xc8&\xb5\'CtR\x14\xedX\xe7\x0fZ\xdar\x02\xe0bK<\xff$\x8b\xecEJ\x15\xc4\xb5\xd5h\x12|a=;\x98\xba\xb9\xf7\xed\xfal\xcf\x1e\xa1x%w\xea\xe0\xb2\xb6\xe5@\x93\x83\x90h_\xb0\x94\x9e{2\xe9""\x84\xc5S_,\xb5i\xb5j,\xe4\x04a\xbe\x9f\xfe4\xac\xff\xe8\x07-R\xdbX\xf7\xdc;\x92;\xd6\xe6\x0f#]\xddeuA\xceJh\xfd\xb1\x85\x04', 'data': 'Minion minion-win-1 started at Wed Apr 10 12:48:56 2024', 'tag': 'minion_start', '_stamp': '2024-04-10T12:48:56.674818'}
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] SaltEvent PUB socket URI: 4510
[DEBUG   ] SaltEvent PULL socket URI: 4511
[DEBUG   ] Sending event: tag = __master_req_channel_payload; data = {'id': 'minion-win-1', 'cmd': '_minion_event', 'pretag': None, 'tok': b'\x19\x94rQG\xa1\xe2"\xc3(z\x90\xa6yV\xcfE\xf5+\x8e\x9e\xb1T\xf5N8+\xd5\xc9&d\xf8\xc0\x85\x8e*$.:L\xd2G\xc3U\n\xea\xd8\xad\xf2}S\xa9]]\xb5\x86\xebCM\xae\x86\xd40\xec\xbaX\x16\x812!\xe3>\xec\xcd\xceq\xd4\x95\xb4\xfbG!\x8f-\xa8]9#\xb7\x08\xcac\xae\xd7\xe6*.\xf5\xca\xa0\xd3\xeaO\xb4\x19\x1a7\xcc DB\x8d\xeaN\x00\x8b\x98C\xccc\xa3\xbc\xf4>\xe0\xad A@\x1fp\xab7\xd8\xfb\xc2\xd1\x1c\xbf\x9a\x03K\xc8&\xb5\'CtR\x14\xedX\xe7\x0fZ\xdar\x02\xe0bK<\xff$\x8b\xecEJ\x15\xc4\xb5\xd5h\x12|a=;\x98\xba\xb9\xf7\xed\xfal\xcf\x1e\xa1x%w\xea\xe0\xb2\xb6\xe5@\x93\x83\x90h_\xb0\x94\x9e{2\xe9""\x84\xc5S_,\xb5i\xb5j,\xe4\x04a\xbe\x9f\xfe4\xac\xff\xe8\x07-R\xdbX\xf7\xdc;\x92;\xd6\xe6\x0f#]\xddeuA\xceJh\xfd\xb1\x85\x04', 'data': 'Minion minion-win-1 started at Wed Apr 10 12:48:56 2024', 'tag': 'salt/minion/minion-win-1/start', '_stamp': '2024-04-10T12:48:56.674818'}
[DEBUG   ] Closing IPCMessageClient instance
[INFO    ] Minion is ready to receive requests!


# Master receives two events
[DEBUG   ] Sending event: tag = salt/minion/minion-win-1/start; data = {'id': 'minion-win-1', 'cmd': '_minion_event', 'pretag': None, 'data': 'Minion minion-win-1 started at Wed Apr 10 12:48:56 2024', 'tag': 'salt/minion/minion-win-1/start', '_stamp': '2024-04-10T12:48:56.901377'}
[DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG   ] Sending event: tag = salt/minion/minion-win-1/start; data = {'id': 'minion-win-1', 'cmd': '_minion_event', 'pretag': None, 'data': 'Minion minion-win-1 started at Wed Apr 10 12:48:56 2024', 'tag': 'salt/minion/minion-win-1/start', '_stamp': '2024-04-10T12:48:56.979531'}

[saltwinqa@saltmaster-pip 3006.7]$ salt-run state.event pretty=true tagmatch='salt/minion/*/start'
salt/minion/minion-win-1/start	{
    "_stamp": "2024-04-10T12:48:56.901377",
    "cmd": "_minion_event",
    "data": "Minion minion-win-1 started at Wed Apr 10 12:48:56 2024",
    "id": "minion-win-1",
    "pretag": null,
    "tag": "salt/minion/minion-win-1/start"
}
salt/minion/minion-win-1/start	{
    "_stamp": "2024-04-10T12:48:56.979531",
    "cmd": "_minion_event",
    "data": "Minion minion-win-1 started at Wed Apr 10 12:48:56 2024",
    "id": "minion-win-1",
    "pretag": null,
    "tag": "salt/minion/minion-win-1/start"
}

Setup
Windows minions 3006.1, 3006.4
The master version seems to be irrelevant.
Does not happen with 3006.6 and 3006.7 minions

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
PASTE HERE

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names no source files, tests, or entry points. Start by reproducing the issue with a Windows minion using a multimaster list and compare versions 3006.1, 3006.4, 3006.6, and 3006.7; done means each master receives one /start event per minion startup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.