OOM while uploading chunks
Open
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 272
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
I'm trying to set up backups on my Nextcloud 23.0.10 instance using this app - version 1.0.4.
Creating an initial full backup manually with ./occ backup:point:create succeeds, as does ./occ backup:point:pack [ID], however once I run ./occ backup:point:upload, it begins uploading all chunks for the backup, and then eventually it runs out of memory, and I have to unlock the backup and then restart the upload. My PHP memory limit is set to 4GB and I'm running this inside Docker, and my external storage is a DigitalOcean Spaces bucket.
Logs:
www-data@8865cf96ff87:~/html$ ./occ backup:point:upload DATE-full-ID --external=16
- uploading DATE-full-ID to external folders
- checking external folder 16:PATH
> restoring point not found
* Creating Restoring Point on external folder: ok
> restoring point created
* initiating health check
> Health status: 0 correct, 541 missing and 0 faulty files
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00001-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00002-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00003-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00004-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00005-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00006-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00007-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00008-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00009-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00010-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00011-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00012-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00013-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00014-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00015-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00016-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00017-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00018-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00019-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00020-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00021-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00022-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00023-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00024-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00025-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00026-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00027-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00028-...: ok
* Uploading data/data-5008c857-0c7f-49bc-a318-.../00029-...: PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 139810256 bytes) in /var/www/html/custom_apps/backup/lib/Service/ExternalFolderService.php on line 262
www-data@8865cf96ff87:~/html$ ./occ backup:point:upload DATE-full-ID --external=16
In MetadataService.php line 155:
point is locked
[...snip...]
www-data@8865cf96ff87:~/html$ ./occ backup:point:unlock DATE-full-ID
www-data@8865cf96ff87:~/html$ ./occ backup:point:upload DATE-full-ID --external=16
- uploading DATE-full-ID to external folders
- checking external folder 16:PATH
> restoring point found
> Health status: 83 correct, 457 missing and 1 faulty files
* Uploading data/data-08152021-9fab-492b-8735-.../00004-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00005-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00006-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00007-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00008-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00009-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00010-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00011-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00012-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00013-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00014-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00015-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00016-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00017-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00018-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00019-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00020-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00021-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00022-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00023-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00024-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00025-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00026-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00027-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00028-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00029-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00030-...: ok
* Uploading data/data-08152021-9fab-492b-8735-.../00031-...: PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 104857648 bytes) in /var/www/html/lib/private/Files/Storage/Common.php on line 208
PHP Fatal error: Uncaught Error: Class "OC\Log\ExceptionSerializer" not found in /var/www/html/lib/private/Log.php:316
Stack trace:
#0 /var/www/html/lib/private/Log/ErrorHandler.php(93): OC\Log->logException(Object(Error), Array)
#1 [internal function]: OC\Log\ErrorHandler::onError(2, 'fopen(/tmp/oc_t...', '/var/www/html/a...', 641)
#2 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(641): fopen('/tmp/oc_tmp_EWe...', 'r')
#3 /var/www/html/apps/files_external/lib/Lib/Storage/AmazonS3.php(501): OCA\Files_External\Lib\Storage\AmazonS3->writeBack('/tmp/oc_tmp_EWe...', 'PATH')
#4 [internal function]: OCA\Files_External\Lib\Storage\AmazonS3->OCA\Files_External\Lib\Storage\{closure}()
#5 /var/www/html/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php(119): call_user_func(Object(Closure))
#6 [internal function]: Icewind\Streams\CallbackWrapper->stream_close()
#7 {main}
thrown in /var/www/html/lib/private/Log.php on line 316
www-data@8865cf96ff87:~/html$
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ExternalFolderService.php around line 262, then compare the later failures in lib/private/Files/Storage/Common.php line 208 and apps/files_external/lib/Lib/Storage/AmazonS3.php. Reproduce ./occ backup:point:upload with the reported external folder and inspect memory use while chunks are uploaded. Done means the upload completes without exhausting PHP memory and the existing restore-point flow remains usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, php
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100