File scans stop when local external storage contains unreadable directories

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Add a local external storage which includes directories with no access permissions for web server user.
2. NC file scan fails at first directory permission denied.

Expected behaviour

Skip directory and carry on.

Actual behaviour

Stops with exception.

Comments

I had noticed that NC stopped updating any file additions/removals to a local external storage shortly after updating from 21.0.4 to 21.0.5 but had also changed the local external storage from an ext4 filesystem mountpoint to an overlayfs lowerdirs only mountpoint (a merged JBOD including the original ext4 filesystem).

Running occ files:scan shows it flaking at a permission denied with the "lost+found" directory. Since "lost+found" directories permission have always been root 700 not sure why it started failing when it was scanning fine before. The overlayfs doesn't cause any user process visible difference and I can't see any obvious related changes in 21.0.5. Bit odd.

As a workaround I've added "lost+found" to the isIgnoredDir list in lib/private/Files/Filesystem.php file but since directory permissions issues on local storage are quite likely: for any local external storage all directory permission failures should be a warning and carry on not an exception and stop.

Reopen of #28277 since it got closed by stale bot without original reporter updating.

Server configuration

Operating system:
Linux
Web server:
Lighttpd
Database:
MariaDB
PHP version:
7.4.25
Nextcloud version: (see Nextcloud admin page)
21.0.5

Where did you install Nextcloud from:
Nextcloud web tar download.

Are you using external storage, if yes which one: local/smb/sftp/...
local

Logs

OCC server error log

$ php occ -vvv files:scan --all
Starting scan for user 1 out of 5 (admin)
        Folder  /admin/
        Folder  /admin/cache
        Folder  /admin/files
        Folder  /admin/files/Media/
        Folder  /admin/files/Media/lost+found/
Exception during scan: opendir(/usr/local/media/lost+found): failed to open dir: Permission denied
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler()
#1 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Local.php(135): opendir()
#2 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Common.php(878): OC\Files\Storage\Local->opendir()
#3 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php(158): OC\Files\Storage\Common->getDirectoryContent()
#4 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php(158): OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#5 [internal function]: OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#6 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(409): iterator_to_array()
#7 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(389): OC\Files\Cache\Scanner->handleChildren()
#8 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(392): OC\Files\Cache\Scanner->scanChildren()
#9 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(341): OC\Files\Cache\Scanner->scanChildren()
#10 /var/www/localhost/htdocs/cloud/lib/private/Files/Utils/Scanner.php(260): OC\Files\Cache\Scanner->scan()
#11 /var/www/localhost/htdocs/cloud/apps/files/lib/Command/Scan.php(158): OC\Files\Utils\Scanner->scan()
#12 /var/www/localhost/htdocs/cloud/apps/files/lib/Command/Scan.php(214): OCA\Files\Command\Scan->scanFiles()
#13 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute()
#14 /var/www/localhost/htdocs/cloud/core/Command/Base.php(169): Symfony\Component\Console\Command\Command->run()
#15 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#16 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#17 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#18 /var/www/localhost/htdocs/cloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run()
#19 /var/www/localhost/htdocs/cloud/console.php(100): OC\Console\Application->run()
#20 /var/www/localhost/htdocs/cloud/occ(11): require_once('/var/www/localh...')
#21 {main}

And repeat for each user sharing the local external storage from /usr/local/media made available under Media in this example.

Start with lib/private/Files/Storage/Local.php at the opendir() call, then trace how lib/private/Files/Storage/Common.php and lib/private/Files/Cache/Scanner.php handle directory contents during occ files:scan. Reproduce the permission-denied case with a local external storage directory. Done means unreadable directories are skipped with a warning and scanning continues.