nextcloud / nextcloud/fulltextsearch

High, permantent CPU load on two MySQL connections

Open
#820 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
234
Forks
64
Avg merge
6h 18m
Merged PRs (30d)
10

Description

We recently updated the host which runs Nextcloud to Ubuntu 22.04 and have some minor issues with the fulltextsearch since then. Not sure if it is only fulltextsearch related or more of a general Nextcloud problem, but you may have more insights here.

After starting/restarting the mysqld service, the CPU load increases after a couple of minutes and then is stuck.

Inspecting the mysqld process with pidstat -t -p PID 1 shows two connections that are responsible for the high load. One at 80%+ and one on 45%+.

15:48:32      UID      TGID       TID    %usr %system  %guest   %wait    %CPU   CPU  Command
[...]
Average:      111         -   2215442   48.10    0.50    0.00    0.00   48.60     -  |__connection
[...]
Average:      111         -   2854935   82.35    0.09    0.00    0.00   82.43     -  |__connection
[...]

Digging into those two threads via mysql> select * from performance_schema.threads where THREAD_OS_ID = 2215442 \G shows two these 2 queries in the PROCESSLIST_INFO field:

TID 2215442
SELECT li.owner_id, li.provider_id, li.document_id, li.collection, li.source, li.status, li.options, li.err, li.message, li.indexed FROM oc_fulltextsearch_index li WHERE (li.status <> '1') AND (err = '0') AND (li.collection = 'local')

and TID 2854935
SELECT size, unencrypted_size FROM oc_filecache WHERE parent = 1149533

Looking into mysql> show processlist; reveals the following:

+-------+-----------------+-----------+-----------+---------+--------+------------------------+------------------------------------------------------------------------------------------------------+
| Id    | User            | Host      | db        | Command | Time   | State                  | Info                                                                                                 |
+-------+-----------------+-----------+-----------+---------+--------+------------------------+------------------------------------------------------------------------------------------------------+
|     5 | event_scheduler | localhost | NULL      | Daemon  | 106103 | Waiting on empty queue | NULL                                                                                                 |
| 20664 | nextcloud       | localhost | nextcloud | Query   |      1 | executing              | SELECT `li`.`owner_id`, `li`.`provider_id`, `li`.`document_id`, `li`.`collection`, `li`.`source`, `l |
| 80561 | nextcloud       | localhost | nextcloud | Query   |      1 | executing              | SELECT `size`, `unencrypted_size` FROM `oc_filecache` WHERE `parent` = 1149533                       |
| 90103 | root            | localhost | NULL      | Query   |      0 | init                   | show processlist                                                                                     |
+-------+-----------------+-----------+-----------+---------+--------+------------------------+------------------------------------------------------------------------------------------------------+

The "Time" field varies between 0 and 3.

While 2215442 sometimes is shortly reduced to a lower CPU usage (prob. because of 'other queries' that are run in between) 2854935 will vary only slightly.

Occasionally another thread opens with this query:
SELECT a.name FROM oc_filecache a LEFT JOIN oc_filecache b ON a.name = b.fileid WHERE (a.storage = '1') AND (b.fileid IS NULL) AND (a.path LIKE 'appdata\\_oc36nkjj747h/preview/_/_/_/_/_/_/_/%') AND (a.mimetype = '2')

After around 30 seconds this query is resolved and the thread reduced to normal/idle.

Running the two 'problematic' queries by hand will return results in less then 3 seconds.

Do you have ideas where the problem could come from and how to fix it?
I can provide more information if needed.

Any help is greatly appreciated.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository files or tests are named. Start by reproducing the sustained load after restarting mysqld, then inspect the listed fulltextsearch and oc_filecache queries with MySQL performance_schema and processlist data. Done means identifying the cause of the persistent CPU usage and documenting a confirmed fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, php
Domain
databases, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.