nextcloud / nextcloud/fulltextsearch

Massive performance improvement on file provider with DB operation under Postgres solving my own issue #799

Open
#850 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
234
Forks
64
Avg merge
6h 18m
Merged PRs (30d)
10

Description

Analysing Postgres performance pointed out a query like this:

An Excerpt!

SELECT "file"."fileid", "storage", "path", "path_hash", "file"."parent", "file"."name", "mimetype", "mimepart", "size", "mtime", "storage_mtime", "encrypted", "etag", "file"."permissions", "checksum", "unencrypted_size", "meta"."json" AS "meta_json", "meta"."sync_token" AS "meta_sync_token" 
FROM "oc_filecache" "file" 
LEFT JOIN "oc_files_metadata" "meta" ON "file"."fileid" = "meta"."file_id" 
WHERE ("file"."name" ILIKE $1) AND ((("storage" = $2) AND (("path" = $3) ........

The relevant index for the ILIKE operation is a compound BTREE index:

CREATE INDEX fs_parent_name_hash ON public.oc_filecache USING btree
(
   parent,
   name
)
;

The relevant index for a LIKE operation in a similar query is a compound BTREE index too:

CREATE INDEX fs_storage_path_prefix ON public.oc_filecache USING btree
(
   storage,
   path
)
;

add the following indexes

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX fs_name_gin_trgm ON public.oc_filecache USING GIN (name gin_trgm_ops);

and

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX fs_path_gin_trgm ON public.oc_filecache USING GIN (path gin_trgm_ops);

My oc_filecache table has over a million entries and the Postgres instance is running on a spare laptop.
The query response is now nearly instantly and on par with the elastic_search file provider response.

May it help those who care!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files or tests; start by locating the file-provider query and the PostgreSQL schema or migration entry points. Reproduce the reported query plan, then assess the proposed pg_trgm indexes and verify that the file-provider operation improves without breaking database setup or existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
database, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.