nextcloud / nextcloud/server

`files:scan-app-data` database optimization

Open
#29,139 7 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage enhancement feature: database feature: filesystem performance 🚀
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

I found something that really speed up files:scan-app-data cron

function getUsersToScan in ./lib/BackgroundJob/ScanFiles.php generarate this SQL:

SELECT MAX(`user_id`) FROM `oc_filecache` `f` INNER JOIN `oc_mounts` `m`  ON `storage_id` = `storage` WHERE `size` < 0 GROUP BY `storage_id`;

but it use fs_size index on oc_filecache, and I think is wrong

Better will work to use fs_storege_size index

For my instance it took about 10 minutes on fs_size index, but on fs_storage_size it takes 2-3 seconds!

So my suggestion is change function to:

private function getUsersToScan(): array {
                $query = $this->connection->getQueryBuilder();
                $query->select($query->func()->max('user_id'))
                        ->from('filecache FORCE INDEX(`fs_storage_size`)', 'f')
                        ->innerJoin('f', 'mounts', 'm', $query->expr()->eq('storage_id', 'storage'))
                        ->where($query->expr()->lt('size', $query->createNamedParameter(0, IQueryBuilder::PARAM_INT)))
                        ->groupBy('storage_id')
                        ->setMaxResults(self::USERS_PER_SESSION);

                return $query->execute()->fetchAll(\PDO::FETCH_COLUMN);
        }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/BackgroundJob/ScanFiles.php at getUsersToScan() and inspect the query builder call and the fs_size and fs_storage_size indexes mentioned in the issue. Run the files:scan-app-data cron against a comparable database and verify that the generated query uses the intended index and improves scan time without changing the returned users.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, sql
Domain
backend, database
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.