nextcloud / nextcloud/server

[Bug]: Allowed memory exhausted when listing folder with a large amount of files

Open
#33,032 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

1. to develop 25-feedback bug feature: database feature: dav feature: filesystem performance 🚀
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

Bug description

When listing the files contained in a folder, the following errors will be triggered if the number of file is big (> ~150 000, depends on the memory limit):

Server log:

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 99152104 bytes) at /var/www/nextcloud/apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68",

Web GUI error:

This directory is unavailable, please check the logs or contact the administrator

The issue mostly exist for the trashbin, but once it exists, the user can't even access its files to clean them up.

Links:

Steps to reproduce
  1. Create a lot of files in a user's files directory: for i in {1..150000}; do echo "New File $i" > file$i.txt; done
  2. Scan those files: occ files:scan --all
  3. Access the user's files with the web UI
Expected behavior

The query should not exhaust the allowed memory by paginating the results.

Nextcloud Logs
{
	"reqId": "ctPxRKNSarVQDP03gQqm",
	"level": 3,
	"time": "2022-06-27T15:27:00+00:00",
	"remoteAddr": "127.0.0.1",
	"user": "admin",
	"app": "PHP",
	"method": "PROPFIND",
	"url": "/remote.php/dav/files/admin/",
	"message": "Allowed memory size of 1073741824 bytes exhausted (tried to allocate 99152104 bytes) at /var/www/nextcloud/apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68",
	"userAgent": "Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36",
	"version": "25.0.0.3",
	"data": {
		"app": "PHP"
	}
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OC\Files\Cache\Cache::getFolderContentsById in lib/private/Files/Cache/Cache.php and the memory failure at apps/dav/lib/Connector/Sabre/PropfindCompressionPlugin.php#68. Reproduce with the listed large-file loop, occ files:scan --all, and a web listing. Done means listing a folder with roughly 150,000 files no longer exhausts the allowed memory, using paginated query results.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.