[Bug]: 🪲 Server side encryption does not encrypt files with S3 primary storage, related error messages on encrypt-all and scan:legacy-format 😢

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Preface

This issue is well reported and discussed around GitHub and the Nextcloud Community forum, but the closest Issue is #33371 but @szaimen said to open a new bug report for ongoing issues which is why I'm creating a new issue for this. I'm hoping this issue gets solved. It seems like Nextcloud has a lot of issues with encryption, especially with S3. Many of these issues were reported months or years ago. 😊

Maybe most people do not use S3 as their primary storage, I'm not sure? S3 is an inexpensive way to store lots of data, especially with providers like Wasabi because it's only $6/TB/month. Hoping that someone takes ownership of this situation, even if that just means preventing new installs from using it or adding something to the manual. 😊

Remaining optimistic!

Background

Using Wasabi as an S3-compataible provider for primary storage in Nextcloud.

After upgrading from Nextcloud 26 or 27, I started receiving errors related to encryption (#8349) and thought I was able to solve it by disabling encryption and re-enabling it, but unfortunately most files remained unencrypted. After hours of troubleshooting, I decided to just start over from scratch.

So I installed a clean Nextcloud instance with a new database, new Redis, everything. But as soon as I set up S3 and enabled server side encryption, I ran into trouble. Files will sync and upload to the S3 provider but are not encrypted, unless they are very small. For example, text files will upload and encrypt, but even an 8 MB zip file will not encrypt. Same result in web client and in Windows client.

Because there are many similar issues, and even with a fresh install issues exists, it seems that there are still some unresolved issues within the codebase when it comes to this type of setup: S3 + server side encryption.

Troubleshooting

After performing the Steps to reproduce and then experiencing the issue, I troubleshooted by running these commands, testing with file uploads after each command:

The logs only appear when running or after running the occ commands. If I do not run the commands, I do not see anything in the log files. The uploads complete as expected without any logging, but the files are not encrypted.

Steps I have tried to resolve:

1. Decrypt all files: sudo -u www-data php occ encryption:decrypt-all
2. Verify encryption disabled: sudo -u www-data php occ encryption:status
3. Disable encryption: sudo -u www-data php occ encryption:disable
4. Scan all files to ensure integrity: sudo -u www-data php occ files:scan --all
5. Uncheck the Encrypt the home storage option in Settings > Security
6. Wait a minute
7. Enable encryption: sudo -u www-data php occ encryption:disable
8. Check the Encrypt the home storage option in Settings > Security
9. Wait a minute
10. Ensure encryption is enabled: sudo -u www-data php occ encryption:status
11. Scan key storage format: occ encryption:scan:legacy-format
12. Migrate key storage format for good measure: sudo -u www-data php occ encryption:migrate-key-storage-format
13. Encrypt all files: sudo -u www-data php occ encryption:encrypt-all

After running that, all of the files are encrypted.... but no new files become encrypted.

Similar issues

Related issues that I have read, implementing fixes and pieces from each:
Server Side Encryption Does Not Encrypt Files When Using S3-Compatible Primary Storage #33371
Encryption not working with S3 object storage as primary storage #11826
Server-side encryption is incompatible with using an object store as a primary data store #22077
Nextcloud update cause encryption files not opened anymore or being encrypted. #2206
Enabled encryption and disabled it again and now it throws me an error in the interface and the logs

Similar family of issues that originally lead me to reinstall Nextcloud:
Encrypt:scan:legacy-format finds files which doesn’t have a proper header in files_version and files_trashbin
Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files #8546
Nextcloud Encryption breaks with OpenSSL 3.x due to legacy RC4 usage #32003
Fix encrypted version to 0 when finding unencrypted file #28373
occ encryption:scan:legacy-format
Nextcloud 20 The old server-side-encryption format is enabled #22478

Many people are simply disabling the "Default Encryption Module" but that is not a solution because it disables encryption by doing that.

One of the issues recommended to implement these:

'encryption.legacy_format_support' => false,
'encryption.key_storage_migrated' => false,

But I had no luck with those.

Steps to reproduce

1. Install Nextcloud (I've seen the issue on both the archive method setup and nextcloud-snap)
2. Attach to an S3-compatible provider (I am using Wasabi)
3. Enable Default Encryption Module in Apps
4. Enable Server Side Encryption and Encrypt the Home Storage (should be default I believe)
5. Upload a small text file
6. Confirm the text file is encrypted on Wasabi
7. Upload a 10 MB file
8. Confirm the file is not encrypted on Wasabi

Expected behavior

Expected behavior is for files to automatically encrypt when uploaded.

Installation method

Community Manual installation with Archive

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "trashbin_retention_obligation": "30, 90",
        "remember_login_cookie_lifetime": 1296000,
        "filelocking.enabled": true,
        "trusted_domains": [
            "localhost",
            "***REDACTED***.com",
        ],
        "lost_password_link": "disabled",
        "overwriteprotocol": "https",
        "htaccess.RewriteBase": "\/",
        "versions_retention_obligation": "7, 30",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "admin"
        ],
        "twofactor_enforced_excluded_groups": [],
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "limit_login_to_ip"
        ],
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "REDACTED.storage",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "s3.us-central-1.wasabisys.com",
                "port": 443,
                "use_ssl": true,
                "region": "us-central-1",
                "use_path_style": false,
                "verify_bucket_exists": false
            }
        },
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0
        },
        "maintenance": false,
        "theme": "",
        "simpleSignUpLink.shown": false,
        "default_phone_region": "us",
        "default_locale": "en_US",
        "default_timezone": "America\/Chicago",
        "defaultapp": "files",
        "loglevel": 2,
        "overwrite.cli.url": "https:\/\/REDACTED.com",
        "data-fingerprint": "***REDACTED***",
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - audioplayer: 3.4.0
  - bruteforcesettings: 2.7.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - encryption: 2.15.0
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - limit_login_to_ip: 4.0.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - theming_customcss: 1.15.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_totp: 9.0.0
  - unsplash: 2.2.1
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - geoblocker: 0.5.11 (installed 0.5.11)
  - survey_client: 1.15.0 (installed 1.15.0)
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Will post below

Additional info

No response

Start by reproducing the issue with an S3-compatible primary store, the Default Encryption Module, and small versus 10 MB uploads. Review the encryption:encrypt-all, encryption:scan:legacy-format, and files:scan entry points, along with linked pull request #57279. Done means uploaded files of both sizes are encrypted and the related commands no longer report inconsistent encryption state.