saltstack / saltstack/salt

inotify not updating watch list with added subdirectories when auto_add and recursive are set to true until restart

Open
#53,290 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Beacon bug Confirmed ZD
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

While using the inotify beacon if you setup a directory to watch and have recursive and auto_add set to true. If you add a subdirectory into the directory being watched new items will not be fully watched. You will get IN_MODIFY noticed for items in the subdirectories but until a restart, you can not get IN_CLOSE_WRITE items or other items of that nature.

once restarted inotify will include the subdirectories in the watched items list so will get the full feature of masks.

This can be further validated by adding a log line just like the following that lists out the watched items

    # Get paths currently being watched
    current = set()
    for wd in wm.watches:
        current.add(wm.watches[wd].path)
        log.debug('current watches: %s',wm.watches[wd].path)
Setup
beacons:
  status:
    - interval: 600
  inotify:
    - files:
        /tmp:
          recurse: True
          mask:
            - close_write
          auto_add: True

restart the minion then add a directory and copy a file into the directory you just created. If inotify had added the subdirectory you made you should see a new event on the bus. but you won't

If you add the logging item about you should see the new subdirectory in the event log if you have debug on. but that also doesn't happen.

After verifying that it works like describe restart the minion and see that all of the above works for that directory.

Steps to Reproduce Issue

restart the minion then add a directory and copy a file into the directory you just created. If inotify had added the subdirectory you made you should see a new event on the bus. but you won't

If you add the logging item about you should see the new subdirectory in the event log if you have debug on. but that also doesn't happen.

After verifying that it works like describe restart the minion and see that all of the above works for that directory.

The following is with the debug log described.

2019-05-28 16:02:17,432 [salt.utils.schedule:35  ][TRACE   ][15073] ==== evaluating schedule now None =====
2019-05-28 16:02:17,771 [salt.beacons     :35  ][TRACE   ][15073] Beacon processing: status
2019-05-28 16:02:17,772 [salt.beacons     :35  ][TRACE   ][15073] Processing interval 600 for beacon mod status
2019-05-28 16:02:17,772 [salt.beacons     :35  ][TRACE   ][15073] Processing interval in map
2019-05-28 16:02:17,772 [salt.beacons     :35  ][TRACE   ][15073] Interval counter: 109
2019-05-28 16:02:17,773 [salt.beacons     :35  ][TRACE   ][15073] Skipping beacon status. Interval not reached.
2019-05-28 16:02:17,773 [salt.beacons     :35  ][TRACE   ][15073] Beacon processing: inotify
2019-05-28 16:02:17,774 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp
2019-05-28 16:02:17,775 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/.Test-unix
2019-05-28 16:02:17,775 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/.ICE-unix
2019-05-28 16:02:17,775 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/.font-unix
2019-05-28 16:02:17,776 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/.X11-unix
2019-05-28 16:02:17,776 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/.XIM-unix
2019-05-28 16:02:17,776 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/openldap-tlsmc-certs--EE74860B999E17892613EA037DAA1B660EDB606E97B1E44773233ED36CCDB728
2019-05-28 16:02:17,776 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/openldap-tlsmc-certs--EE74860B999E17892613EA037DAA1B660EDB606E97B1E44773233ED36CCDB728/cacerts
2019-05-28 16:02:17,777 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-chronyd.service-OCIMwl
2019-05-28 16:02:17,777 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-chronyd.service-OCIMwl/tmp
2019-05-28 16:02:17,777 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-named.service-dW6n5D
2019-05-28 16:02:17,777 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][15073] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-named.service-dW6n5D/tmp
2019-05-28 16:02:17,778 [salt.utils.schedule:35  ][TRACE   ][15073] ==== evaluating schedule now None =====

after a restart the following

2019-05-28 16:06:54,560 [salt.utils.schedule:35  ][TRACE   ][16271] ==== evaluating schedule now None =====
2019-05-28 16:06:55,554 [salt.beacons     :35  ][TRACE   ][16271] Beacon processing: status
2019-05-28 16:06:55,554 [salt.beacons     :35  ][TRACE   ][16271] Processing interval 600 for beacon mod status
2019-05-28 16:06:55,555 [salt.beacons     :35  ][TRACE   ][16271] Processing interval in map
2019-05-28 16:06:55,555 [salt.beacons     :35  ][TRACE   ][16271] Interval counter: 4
2019-05-28 16:06:55,555 [salt.beacons     :35  ][TRACE   ][16271] Skipping beacon status. Interval not reached.
2019-05-28 16:06:55,555 [salt.beacons     :35  ][TRACE   ][16271] Beacon processing: inotify
2019-05-28 16:06:55,557 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp
2019-05-28 16:06:55,557 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/.Test-unix
2019-05-28 16:06:55,557 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/.ICE-unix
2019-05-28 16:06:55,558 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/.font-unix
2019-05-28 16:06:55,558 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/.X11-unix
2019-05-28 16:06:55,558 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/.XIM-unix
2019-05-28 16:06:55,558 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/openldap-tlsmc-certs--EE74860B999E17892613EA037DAA1B660EDB606E97B1E44773233ED36CCDB728
2019-05-28 16:06:55,558 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/openldap-tlsmc-certs--EE74860B999E17892613EA037DAA1B660EDB606E97B1E44773233ED36CCDB728/cacerts
2019-05-28 16:06:55,559 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-chronyd.service-OCIMwl
2019-05-28 16:06:55,559 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-chronyd.service-OCIMwl/tmp
2019-05-28 16:06:55,559 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-named.service-dW6n5D
2019-05-28 16:06:55,559 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/systemd-private-7eff4f142f7043ea8ec6b350b31e1089-named.service-dW6n5D/tmp
2019-05-28 16:06:55,560 [salt.loaded.ext.beacons.inotify:275 ][DEBUG   ][16271] current watches: /tmp/verify
Versions Report
    Salt Version:
               Salt: 2018.3.4
     
    Dependency Versions:
               cffi: 1.6.0
           cherrypy: Not Installed
           dateutil: 2.6.0
          docker-py: Not Installed
              gitdb: Not Installed
          gitpython: Not Installed
              ioflo: Not Installed
             Jinja2: 2.7.2
            libgit2: Not Installed
            libnacl: Not Installed
           M2Crypto: Not Installed
               Mako: Not Installed
       msgpack-pure: Not Installed
     msgpack-python: 0.4.6
       mysql-python: Not Installed
          pycparser: 2.14
           pycrypto: 2.6.1
       pycryptodome: Not Installed
             pygit2: Not Installed
             Python: 2.7.14 (default, Jul 26 2018, 19:59:38)
       python-gnupg: Not Installed
             PyYAML: 3.10
              PyZMQ: 15.3.0
               RAET: Not Installed
              smmap: Not Installed
            timelib: Not Installed
            Tornado: 4.2.1
                ZMQ: 4.1.4
     
    System Versions:
               dist:   
             locale: ascii
            machine: x86_64
            release: 4.14.104-95.84.amzn2.x86_64
             system: Linux
            version: Not Installed

And

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.11.5
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: 1.6.1
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Apr  9 2019, 14:30:50)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.12.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the inotify beacon implementation identified by the salt.loaded.ext.beacons.inotify debug path and reproduce the supplied configuration with recursive and auto_add enabled. Inspect the watched paths and emitted masks after creating a subdirectory, then verify that newly added directories receive the same event handling without a restart. Done means the supplied reproduction produces the expected close_write event and updates the watch list immediately.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.