[Bug]: Allowed memory exhausted when listing folder with a large amount of files
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
Bug description
When listing the files contained in a folder, the following errors will be triggered if the number of file is big (> ~150 000, depends on the memory limit):
Server log:
Allowed memory size of 1073741824 bytes exhausted (tried to allocate 99152104 bytes) at /var/www/nextcloud/apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68",
Web GUI error:
This directory is unavailable, please check the logs or contact the administrator
The issue mostly exist for the trashbin, but once it exists, the user can't even access its files to clean them up.
Links:
Steps to reproduce
- Create a lot of files in a user's files directory:
for i in {1..150000}; do echo "New File $i" > file$i.txt; done - Scan those files:
occ files:scan --all - Access the user's files with the web UI
Expected behavior
The query should not exhaust the allowed memory by paginating the results.
Nextcloud Logs
{
"reqId": "ctPxRKNSarVQDP03gQqm",
"level": 3,
"time": "2022-06-27T15:27:00+00:00",
"remoteAddr": "127.0.0.1",
"user": "admin",
"app": "PHP",
"method": "PROPFIND",
"url": "/remote.php/dav/files/admin/",
"message": "Allowed memory size of 1073741824 bytes exhausted (tried to allocate 99152104 bytes) at /var/www/nextcloud/apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68",
"userAgent": "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36",
"version": "25.0.0.3",
"data": {
"app": "PHP"
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with OC\Files\Cache\Cache::getFolderContentsById in lib/private/Files/Cache/Cache.php and the memory failure at apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68. Reproduce with the listed large-file loop, occ files:scan --all, and a web listing. Done means listing a folder with roughly 150,000 files no longer exhausts the allowed memory, using paginated query results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100