[Bug]: `files_encryption/keys/files` contains orphaned key of already deleted files

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

One user historically had a few thousand files in their share.
They were deleted.
A handful of files remain, but still I get the following

occ files:scan -- user.name
Starting scan for user 1 out of 1 (user.name)
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 2852    | 3579  | 0   | 0       | 0       | 0      | 00:00:44     |
+---------+-------+-----+---------+---------+--------+--------------+

All these files seem to live in files_encryption/keys/files.

Tried occ files:cleanup as well to no avail.

This is somewhat related to https://help.nextcloud.com/t/files-encryption-folder-full-of-orphaned-keys/24142, but encryption is still on in my case.

Internal ticket 14987

Steps to reproduce

Unsure how or why this happened, the affected/deleted files are partially from 2018 and some are only a few weeks old.
So I guess:

1. Have encryption enabled since NC17 or so.
2. Do updates until NC30.
3. Upload files
4. Delete them later
5. Delete them from trash bin as well.
6. See their keys still on disk.

However, if I test this with some other user, the files get deleted as they should. I cannot really tell how to reproduce this exact behavior.

Expected behavior

Deleting files should also delete their corresponding keys.

Can someone point me in the right direction to debug which process should clean this up and why it does not do it?

I guess a process that takes care of cleaning up orphaned files is needed when - for whichever reason - the deletion does not happen.
I can do this using a simple shell script, but I am unsure whether there might be DB entries as well.

If anything makes the deletion fail initially, a DB entry needs to be set so it can be cleaned up later on, I'd say.

Nextcloud Server version

30

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 28 to 29)

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "config_is_read_only": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/",
        "htaccess.RewriteBase": "\/",
        "dbtype": "mysql",
        "version": "30.0.1.2",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": false,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": "true",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": ***REMOVED SENSITIVE VALUE***,
            "timeout": 0
        },
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "localstorage.umask": 7,
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "log_type": "file",
        "log_rotate_size": 10000000,
        "loglevel": 2,
        "logfile": "***REMOVED SENSITIVE VALUE***\/nextcloud.log",
        "logfilemode": 400,
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "log_type_audit": "file",
        "log_rotate_size_audit": 10000000,
        "logfile_audit": "***REMOVED SENSITIVE VALUE***\/audit.log",
        "logfilemode_audit": 400,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "mail_smtpstreamoptions": {
            "ssl": {
                "allow_self_signed": true,
                "verify_peer": false,
                "verify_peer_name": false
            }
        },
        "maintenance": false,
        "updater.release.channel": "stable",
        "trashbin_retention_obligation": "auto, 14",
        "versions_retention_obligation": "auto, 60",
        "mail_sendmailmode": "smtp",
        "app_install_overwrite": [
            "impersonate",
            "spreed",
            "files_accesscontrol",
            "quota_warning",
            "richdocuments"
        ],
        "encryption.legacy_format_support": false,
        "encryption.key_storage_migrated": false,
        "defaultapp": "files",
        "customclient_desktop": ""
    }
}

List of activated Apps

Enabled:
  - activity: 3.0.0
  - admin_audit: 1.20.0
  - app_api: 4.0.0
  - bruteforcesettings: 3.0.0
  - calendar: 5.0.1
  - cloud_federation_api: 1.13.0
  - comments: 1.20.1
  - contactsinteraction: 1.11.0
  - dav: 1.31.1
  - encryption: 2.18.0
  - federatedfilesharing: 1.20.0
  - federation: 1.20.0
  - files: 2.2.0
  - files_accesscontrol: 1.20.1
  - files_downloadlimit: 3.0.0
  - files_pdfviewer: 3.0.0
  - files_reminders: 1.3.0
  - files_sharing: 1.22.0
  - files_trashbin: 1.20.1
  - firstrunwizard: 3.0.0
  - groupquota: 0.2.1
  - impersonate: 1.17.0
  - logreader: 3.0.0
  - lookup_server_connector: 1.18.0
  - nextcloud_announcements: 2.0.0
  - notifications: 3.0.0
  - oauth2: 1.18.1
  - password_policy: 2.0.0
  - photos: 3.0.2
  - privacy: 2.0.0
  - provisioning_api: 1.20.0
  - quota_warning: 1.20.0
  - recommendations: 3.0.0
  - related_resources: 1.5.0
  - serverinfo: 2.0.0
  - settings: true
  - sharebymail: 1.20.0
  - text: 4.1.0
  - theming: 2.5.0
  - theming_customcss: 1.17.0
  - twofactor_backupcodes: 1.19.0
  - updatenotification: 1.20.0
  - user_ldap: 1.21.0
  - viewer: 3.0.0
  - webhook_listeners: 1.1.0-dev
  - workflowengine: 2.12.0
Disabled:
  - circles: 30.0.0-dev (installed 26.0.1)
  - dashboard: 7.10.0 (installed 7.0.0)
  - files_downloadactivity: 1.17.0 (installed 1.17.0)
  - files_external: 1.22.0 (installed 1.11.1)
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - files_trackdownloads: 1.11.0 (installed 1.11.0)
  - files_versions: 1.23.0 (installed 1.16.0)
  - spreed: 20.0.1 (installed 20.0.1)
  - support: 2.0.0 (installed 1.6.0)
  - survey_client: 2.0.0 (installed 1.11.0)
  - suspicious_login: 8.0.0
  - systemtags: 1.20.0 (installed 1.13.0)
  - twofactor_nextcloud_notification: 4.0.0
  - twofactor_totp: 12.0.0-dev
  - user_status: 1.10.0 (installed 1.0.1)
  - weather_status: 1.10.0 (installed 1.0.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No errors whatsoever during a files:scan run.

Additional info

No response

Start by tracing the encryption cleanup behavior behind occ files:scan and occ files:cleanup, focusing on files_encryption/keys/files and the corresponding file records. Compare deletion and trash-bin cleanup for affected files with the case that reproduces correctly. Done means deleted files no longer leave orphaned keys, with a cleanup path for failures.