saltstack / saltstack/salt

[BUG] FileNotFoundError for minion key on master causes master abnormal behavior

Open
#62,884 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
two masters (3004.2) sharing via NFS mount point /etc/salt/pki/master folder
windows (and rhel) minions (3004.2) identically configured in multimaster setup
master:

  • master-1
  • master-2

autosign_grains option is configured and working ok, minions are automatically accepted

There's a NFS delay (in the /etc/salt/pki/master) that causes a sync delay when a file is deleted.
It's a random delay between 2 seconds and 10 seconds.
So when a minion key is deleted, it takes a few seconds for the second master (the master from which the key was not deleted) to see the change.

Under this condition the following problem is seen:

  • Stop the minion-win-2 for which the key is going to be deleted
  • Deleted minion-win-2 key from master-1 (so the minion key file is no longer present in /etc/salt/pki/master)
    Immediately (master-2 still believes the file is present)
  • Start the minion (timing is crucial, it has to start before master-2 has updated /etc/salt/pki/master, this server still looks like having the file that was deleted from master-1)
  • All seems to be good on master-1
  • But Master-2 throws error and become unresponsive, restart required
  • Attempts to test.ping from master-2 failed
  • Random responses for salt \* test.ping from master-2 (minion-win-2 always fails, but also randomly other minions fails to respond). No issues on master-1
  • Looking at the minion-win-2, the minion is unable to establish connection with master-2
  • Sooner or later on master-2 a salt time-out is seen when running salt \* test.ping

The error on master-2 is the following:

[DEBUG   ] Sending event: tag = salt/job/20221013200434076446/ret/minion-win-2; data = {'cmd': '_return', 'id': 'minion-win-2', 'success': True, 'return': True, 'retcode': 0, 'jid': '20221013200434076446', 'fun': 'test.ping', 'fun_args': [], '_stamp': '2022-10-13T20:04:35.510899'}
[INFO    ] Authentication request from minion-win-2
[ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7f8240ceb0b8> exception was never retrieved: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/lib/python3.6/site-packages/salt/transport/zeromq.py", line 781, in handle_message
    stream.send(salt.payload.dumps(self._auth(payload["load"], sign_messages)))
  File "/usr/lib/python3.6/site-packages/salt/transport/mixins/auth.py", line 287, in _auth
    with salt.utils.files.fopen(pubfn, "r") as pubfn_handle:
  File "/usr/lib/python3.6/site-packages/salt/utils/files.py", line 385, in fopen
    f_handle = open(*args, **kwargs)  # pylint: disable=resource-leakage
FileNotFoundError: [Errno 2] No such file or directory: '/etc/salt/pki/master/minions/minion-win-2'

To fix it, master-2 master service needs to be restarted.

As per the error thrown, and the mentioned conditions, master-2 thinks the minion key file is still present '/etc/salt/pki/master/minions/minion-win-2', but it's not in real (it's just a matter of giving it a few more seconds to updated the nfs shared mount point)

The real problem here, despite this being probably sort of an edge case, is that the server requires to be restarted.
Even worst, looking at the master-2 log, despite the mentioned error, everything else seems "NORMAL"

From this moment on, this master shows erratic beavhior, (random) minions not responding, time out, etc [root@saltmaster-2 ~]# salt \* test.ping [ERROR ] Message timed out Salt request timed out. The master is not responding. You may need to run your command with --async` in order to bypass the congested event bus.

Minion:

2022-10-13 19:28:12,312 [salt.transport.zeromq:1237][DEBUG   ][6104] Trying to connect to: tcp://172.21.0.5:4506
...
[root@saltmaster-2 ~]# salt \* test.ping
master-2-minion:
    True
minion-rhel7-1:
    True
master-1-minion:
    True
minion-win-2:
    True
minion-win-1:  #
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20221013180321304046


[root@saltmaster-2 ~]# salt \* test.ping
master-2-minion:
    True
minion-rhel7-1:
    True
master-1-minion:
    True
minion-win-1:
    True
minion-win-2:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20221013180615659712

[root@saltmaster-2 ~]# salt \* test.ping
[ERROR   ] Message timed out
Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

Setup
as described in the description
multimaster, with nfs share for /etc/salt/pki/master and autosign_grains working

Steps to Reproduce the behavior
as described, timing is crucial, if enough time is given betweek the key deletion and the minion start to allow the master-2 to reflect the change in its own mount point for /etc/salt/pki/master the issue is not triggered.

Expected behavior
Master service should response as normal, this issue should not cause such a problem

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report
3004.2

Additional context
autosign_grains working

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the multimaster race using the shared /etc/salt/pki/master NFS mount, deleting a minion key and authenticating before the other master sees the deletion. Start with salt/transport/zeromq.py handle_message and salt/transport/mixins/auth.py _auth, which opens the key through salt/utils/files.py; done means the master remains responsive after the missing-key error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.