saltstack / saltstack/salt

race condition in state.file _check_directory for transient files

Open
#65,124 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
A race condition exists in _check_directory where a file is removed after the walk_l is generated but the file is tested with the file.stats . While https://github.com/saltstack/salt/pull/36928 handles a similar case this race condition is different. It is harder to reproduce since the timing is smaller.

Setup
Using this state we occasionally observe (less than 1% occurrence) the failure (stack trace below)

m_dir_perms:
  file.directory:
    - mode: '0755'
    - user: user1
    - group: user1
    - recurse:
       - user
       - group
    - names:
       - /opt/m
  • [X ] on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • classic packaging

Steps to Reproduce the behavior

  1. Setup mongo with data directory of /opt/m
  2. Setup salt with the above state to ensure user/group recursive permissions on the directory.

Expected behavior
All files in the tree have ownership correctly set. If a transient file exists, it is not reported as a failure since it was removed.

Stack Trace
I have adjusted the file name to remove my specific file.

File \"/usr/lib/python3/dist-packages/salt/states/file.py\", line 756, in _check_directory
 stats = __salt__[\"file.stats\"](path, None, follow_symlinks)
File \"/usr/lib/python3/dist-packages/salt/loader/lazy.py\", line 149, in __call__
 return self.loader.run(run_func, *args, **kwargs)
File \"/usr/lib/python3/dist-packages/salt/loader/lazy.py\", line 1228, in run
  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File \"/usr/lib/python3/dist-packages/salt/loader/lazy.py\", line 1243, in _run_as
  return _func_or_method(*args, **kwargs)
File \"/usr/lib/python3/dist-packages/salt/modules/file.py\", line 4094, in stats
   raise CommandExecutionError(\"Path not found: {}\".format(path))
   salt.exceptions.CommandExecutionError: Path not found: /opt/m/a/b/c",

Versions Report

Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.10
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.1

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

Additional context
As noted it is hard to reproduce. On last ~150k highstates the error was observed 4 times.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/states/file.py at _check_directory and inspect the salt/modules/file.py stats entry point shown in the stack trace. Reproduce with the provided file.directory state and a transient file under /opt/m if possible. Done means removed files do not produce a Path not found failure while ownership is still corrected for files that remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.