nextcloud / nextcloud/server

Use MySQL/MariaDB fulltext indexing on oc_filecache to speed up search results

Open
#15,300 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

1. to develop enhancement feature: database feature: filesystem performance 🚀
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

Steps to reproduce
  1. Run Nextcloud on a modern multi-core 64 bit machine with more than a few 100 000 files.
  2. Try and search for a file using the normal search function (no apps/add-ons).
Expected behaviour

Nextcloud should return a set of results in a timely manner (less than 10 seconds ideally).

Actual behaviour

Nextcloud never returns the results.
Underneath, the MySQL/MariaDB server utilizes 100% on one of the server's CPU cores for some time (more than a minute).

Server configuration

Ubuntu 18.04 running Nextcloud 16 Docker Hub image.
MariaDB 10.3.12 (Docker Hub) running off a RAM drive.

Cause

In lib/private/Files/Cache/Cache.php, line 646:

// Original
$sql = ’
SELECT fileid, storage, path, parent, name,
mimetype, storage_mtime, mimepart, size, mtime,
encrypted, etag, permissions, checksum
FROM *PREFIX*filecache
WHERE storage = ? AND name ILIKE ?’;

This query doesn't use an index AFAIK, it scans the entire table, which is why it is slow.

Adding a fulltext index and modifying the query as below significantly reduces the query time and makes the search function usable :

// MariaDB/MySQL FULLTEXT Index
$sql = ’
SELECT fileid, storage, path, parent, name,
mimetype, storage_mtime, mimepart, size, mtime,
encrypted, etag, permissions, checksum
FROM *PREFIX*filecache
WHERE storage = ? AND MATCH(name) AGAINST (?)’;

I don't fully understand how Nextcloud's database schema migration works (the above requires a fulltext index). The fulltext index can be added like so :
ALTER TABLE oc_filecache ADD FULLTEXT INDEX name_idx (name);

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read lib/private/Files/Cache/Cache.php around line 646 and investigate Nextcloud’s database schema migration process for the required MySQL/MariaDB full-text index. Done means the file search uses the proposed indexed lookup where supported, the index is created through the migration system, and searches return results without the reported prolonged database scan.

Written by the indexing model from the issue text.

Assessment

Tech stack
mariadb, mysql, php
Domain
backend, databases, performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.