[Bug]: Extremely slow trashbin restore
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
⚠️ This issue respects the following points: ⚠️
- This is a bug, not a question or a configuration/webserver/proxy issue.
- This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
- Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- I agree to follow Nextcloud's Code of Conduct.
Bug description
Upon synchronization my Windows share I made an error in the exclude list..
That caused the Linux server to become slow. This triggered me in realizing the faulty sync. Then some 40.000 files were already deleted!
I went to the web-interface to start restoring.. and discovered that that also was an extremely server intensive activity.
Steps to reproduce
- Delete many thousands of files on your Windows share
- Restore them via the web-interface
- See the "show full processlist;"
Expected behavior
Since I use AWS-S3 I expected the restore to be a simple MySql update of the folder entry..
Nextcloud Server version
30
Operating system
RHEL/CentOS
PHP engine version
PHP 8.1
Web server
Apache (supported)
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
None
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
- Default user-backend (database)
- LDAP/ Active Directory
- SSO - SAML
- Other
Configuration report
-
List of activated Apps
-
Nextcloud Signing status
-
Nextcloud Logs
-
Additional info
With the command "show full processlist;"
I saw that these were the queries I saw several of that were running a multitude of seconds.
SELECT filecache.fileid, storage, path, path_hash, filecache.parent, filecache.name, mimetype, mimepart, size, mtime
, storage_mtime, encrypted, etag, filecache.permissions, checksum, unencrypted_size, metadata_etag, creation_time, upload_time
, meta.json AS meta_json, meta.sync_token AS meta_sync_token FROM oc_filecache filecache
LEFT JOIN oc_filecache_extended fe ON filecache.fileid = fe.fileid
LEFT JOIN oc_files_metadata meta ON filecache.fileid = meta.file_id
WHERE (filecache.parent = 88189) AND (storage = 2) ORDER BY name ASC
id 88189 is the ID of the trashbin. So it appears that after each delete the new remaining lis of files in the Trashbin is read.
But with my still some 40.000 files remaining in the trashbin this is a big query! I expect things would dramaticaly spead up when the MOVE / RESTORE function become separate from the "show trashbin content".
In other words, why not just first simply RESTORE everything I told it to.. and then.. all at the end read what's left in the trash?
If that could be done, the operations in the trashbin will likely bee only slighly slower when restoring just one file (since the restore call will be split into two operations). But when the RESTORE (or the permanently delete?) action is done on a multitude of files I expect the response will be far faster and the load on the server far lower!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the web-interface trashbin restore flow and the filecache query shown in the issue, using the reported trashbin parent ID and MariaDB process list as the reproduction signals. Compare restoring many files with the trashbin contents being refreshed after each operation. Done means bulk restore no longer repeatedly reads the full remaining trashbin list, with the reported server load and query behavior improved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, mariadb, php
- Domain
- backend, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100