[Bug]: `occ files:copy` fails with large files on S3

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

occ files:copy exception while copying large files or folders

Steps to reproduce

1. Mount an external S3 storage
2. Create a group folder
3. Run occ copy:files to copy a large file or folder from the external storage to group folder

Expected behavior

External storage file copied to group folder

Nextcloud Server version

30

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 28 to 29)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 4,
        "default_phone_region": "BR",
        "trusted_domains": [
            "localhost",
            "drive.ct.utfpr.edu.br"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "skeletondirectory": "\/var\/lib\/nextcloud\/skeleton",
        "templatedirectory": "\/var\/lib\/nextcloud\/templates",
        "dbtype": "pgsql",
        "version": "30.0.0.14",
        "overwrite.cli.url": "https:\/\/drive.ct.utfpr.edu.br\/",
        "htaccess.RewriteBase": "\/",
        "overwriteprotocol": "https",
        "overwritehost": "drive.ct.utfpr.edu.br",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "logfile": "\/var\/lib\/nextcloud\/data\/nextcloud.log",
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "drive.ct.utfpr.edu.br",
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "use_ssl": true,
                "region": "sa-east-1",
                "use_path_style": false
            }
        },
        "theme": "",
        "loglevel": 2,
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "PLAIN",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": true,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "filelocking.enabled": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***",
            "dbindex": 0
        },
        "memcached_servers": [
            [
                "cache.cogeti.ct.internal",
                11211
            ]
        ],
        "diagnostics.logging": true,
        "diagnostics.logging.threshold": 0
    }
}

List of activated Apps

Enabled:
  - activity: 3.0.0
  - admin_audit: 1.20.0
  - bruteforcesettings: 3.0.0
  - circles: 30.0.0-dev
  - cloud_federation_api: 1.13.0
  - comments: 1.20.1
  - contactsinteraction: 1.11.0
  - dashboard: 7.10.0
  - dav: 1.31.1
  - federatedfilesharing: 1.20.0
  - federation: 1.20.0
  - files: 2.2.0
  - files_downloadlimit: 3.0.0
  - files_external: 1.22.0
  - files_pdfviewer: 3.0.0
  - files_reminders: 1.3.0
  - files_sharing: 1.22.0
  - files_trashbin: 1.20.1
  - files_versions: 1.23.0
  - firstrunwizard: 3.0.0
  - groupfolders: 18.0.1
  - logreader: 3.0.0
  - lookup_server_connector: 1.18.0
  - nextcloud_announcements: 2.0.0
  - notifications: 3.0.0
  - oauth2: 1.18.1
  - password_policy: 2.0.0
  - photos: 3.0.2
  - privacy: 2.0.0
  - provisioning_api: 1.20.0
  - recommendations: 3.0.0
  - related_resources: 1.5.0
  - serverinfo: 2.0.0
  - settings: 1.13.0
  - sharebymail: 1.20.0
  - support: 2.0.0
  - survey_client: 2.0.0
  - systemtags: 1.20.0
  - text: 4.1.0
  - theming: 2.5.0
  - twofactor_backupcodes: 1.19.0
  - updatenotification: 1.20.0
  - user_ldap: 1.21.0
  - user_oidc: 6.0.1
  - user_status: 1.10.0
  - viewer: 3.0.0
  - weather_status: 1.10.0
  - webhook_listeners: 1.1.0-dev
  - workflowengine: 2.12.0
Disabled:
  - encryption: 2.18.0
  - suspicious_login: 8.0.0
  - twofactor_nextcloud_notification: 4.0.0
  - twofactor_totp: 12.0.0-dev

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

I am transferring nextcloud to another server. The new server is configured to use Amazon S3 as primary storage. The other server has 63 S3 external mounts. Each of those S3 mounted folders are to be copied to a corresponding group folder on the new server. As I cannot simply copy files to the new server because S3 primary storage uses a completely different object structure I am trying several copy strategies.

The buckets on the old server are in the same account and region as the destination primary bucket. I am trying the occ files:copy command. The copy process is very fast but unfortunately the files:copy aborts the transfer for large (it seem that it cannot copy more than 5GB) files or folders.

The old server is running Nextcloud 29.0.5 and the destination is 30.0.0.

Below is an example of what happens when I try to copy a 5GB file (S3 mount) from the old server to the new server (group folder):

$ occ files:copy -vvv /cogeti/files/TESTE-COTED-CT/eLearningSuite_6_1_LS12.7z /cogeti/files/COTED-CT/

In Node.php line 412:

  [OCP\Files\NotPermittedException]
  Could not copy /cogeti/files/TESTE-COTED-CT/eLearningSuite_6_1_LS12.7z to /cogeti/files/COTED-CT/eLearningSuite_6_1_LS12.7z

Below is what happens when I try copy a 5GB file from the

Exception trace:
  at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/lib/private/Files/Node/Node.php:412
 OC\Files\Node\Node->copy() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/apps/files/lib/Command/Copy.php:112
 OCA\Files\Command\Copy->execute() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/3rdparty/symfony/console/Command/Command.php:298
 Symfony\Component\Console\Command\Command->run() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/3rdparty/symfony/console/Application.php:1040
 Symfony\Component\Console\Application->doRunCommand() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/3rdparty/symfony/console/Application.php:301
 Symfony\Component\Console\Application->doRun() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/3rdparty/symfony/console/Application.php:171
 Symfony\Component\Console\Application->run() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/lib/private/Console/Application.php:183
 OC\Console\Application->run() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/console.php:87
 require_once() at /efs/ecs/drive.ct.utfpr.edu.br/nextcloud/occ:11

While copying I see the eLearningSuite_6_1_LS12.7z.part file inside the group folder. The file disappears after the copy aborts.

I was able to copy a 2.9GB file between the very same folders in a few seconds but the 5GB file throws the OCP\Files\NotPermittedException exception. The user has full permissions on both folders and has an unlimited quota so it cannot be permission or quota related.

The same happens when I try to copy an entire folder with more than 5GB total files, even if there are not very large files in the origin folder.

I have also tried to use the GUI an DAV (via curl). In both cases the copy aborts after a few seconds. DAV aborts with a 504 gateway timeout status. BTW, as expected, DAV copy is much slower than the files:copy command.

Start with the occ files:copy entry point in apps/files/lib/Command/Copy.php and the failure reported at lib/private/Files/Node/Node.php:412. Reproduce the copy using an S3 external mount and a group folder with files below and above 5GB, then trace the NotPermittedException. Done means large files and folders copy successfully without the partial .part file disappearing.