[Bug]: DecryptionFailedException for user-key encryption when generating blurhash metadata

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When running NextCloud 29.0.4 with SSE via user-key encryption (occ encryption:disable-master-key), the error

DecryptionFailedException Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.

is logged. The log easily builds up to > 100 MB. It is unclear if the cron job finishes successfully or stop due to this error.

As one can reproduce with the steps below (see also the stack traces below), the following happens:

First, a user-independent process (like a cronjob (see Method A below) or occ run manually (see Method B below)) triggers the update of metadata. This is fine for file types that do not support any preview.

But for files with preview support, GenerateBlurhashMetadata requests the preview of this file by calling getPreview. Since this new file does not have any preview yet, the preview generator tries to create one. For this, it needs to read the original file. But since user-specific key encryption is enabled, the file can only be read from a user session. We, however, are trying to update the blurhash outside of a user session and therefore fail to decrypt the file. No preview can be generated and no blurhash calculated.

For user-specific encryption, each file access to encrypted data needs to fail softly, which is not the case here. Below, two methods how to reproduce are shown but probably even more exist that call the getPreview outside a user session.

Steps to reproduce

Preparation

1. Have NextCloud 29 installed with user-key encryption (occ encryption:disable-master-key) enabled.
2. Create a new file test/test.txt (it is important that it is a file format compatible with Preview like a text file or image).

There are two known ways to reproduce the error.

Method A (Cron Job)

1. Make sure you have background jobs activated via crontabs (i.e., independent of a user session), e.g.,*/5 * * * * php -f /var/www/nextcloud/cron.php in crontab.
2. Edit contents of the test file.
3. Wait for the cron job to start.
4. See the error message in the logs.

Method B (Force Generation of Metadata)

1. Run occ files:scan --path=<username>/files/test -vvv and notice that it works without errors.
2. Run occ files:scan --path=<username>/files/test -vvv --generate-metadata and notice that it fails with the mentioned error.

Expected behavior

• No error message.
• Background job finishes sucessfully.
• occ files:scan works with --generate-metadata

Installation method

None

Nextcloud Server version

29

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Enabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "forcessl": true,
        "theme": "",
        "maintenance": false,
        "secret": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 2,
        "trashbin_retention_obligation": "auto",
        "updater.release.channel": "stable",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "app_install_overwrite": [
            "calendar"
        ],
        "mysql.utf8mb4": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "default_phone_region": "DE",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 1,
        "preview_max_x": 1024,
        "preview_max_y": 1024
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.15
  - cfg_share_links: 5.1.1
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - dav: 1.30.1
  - encryption: 2.17.0
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - notes: 4.10.1
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - support: 1.12.0
  - survey_client: 1.17.0
  - systemtags: 1.19.0
  - text: 3.10.1
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - contactsinteraction: 1.10.0 (installed 1.5.0)
  - dashboard: 7.9.0 (installed 7.0.0)
  - end_to_end_encryption: 1.15.2 (installed 1.15.2)
  - files_external: 1.21.0
  - files_trashbin: 1.19.0 (installed 1.19.0)
  - richdocuments: 8.4.4 (installed 8.4.4)
  - suspicious_login: 7.0.0
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Method A:
{"reqId":"I5UTgbZ4ReDZqoiuJFBD","level":2,"time":"2024-08-16T17:10:03+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"issue while running UpdateSingleMetadata","userAgent":"--","version":"29.0.4.1","exception":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Stream/Encryption.php","line":517,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Stream/Encryption.php","line":316,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->"},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->"},{"file":"/var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread"},{"file":"/var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php","line":96,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->"},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/TXT.php","line":65,"function":"stream_get_contents"},{"file":"/var/www/nextcloud/lib/private/Preview/GeneratorHelper.php","line":64,"function":"getThumbnail","class":"OC\\Preview\\TXT","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":361,"function":"getThumbnail","class":"OC\\Preview\\GeneratorHelper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":337,"function":"generateProviderPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":143,"function":"getMaxPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":110,"function":"generatePreviews","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/PreviewManager.php","line":190,"function":"getPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/Blurhash/Listener/GenerateBlurhashMetadata.php","line":90,"function":"getPreview","class":"OC\\PreviewManager","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":86,"function":"handle","class":"OC\\Blurhash\\Listener\\GenerateBlurhashMetadata","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":230,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":59,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":86,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":98,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":114,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/FilesMetadata/Job/UpdateSingleMetadata.php","line":60,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/Job.php","line":80,"function":"run","class":"OC\\FilesMetadata\\Job\\UpdateSingleMetadata","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":61,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":47,"function":"start","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":177,"function":"execute","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"}],"File":"/var/www/nextcloud/apps/encryption/lib/Crypto/Encryption.php","Line":340,"Hint":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","message":"issue while running UpdateSingleMetadata","exception":[],"userId":"<redacted>","fileId":504692,"CustomMessage":"issue while running UpdateSingleMetadata"},"id":"66bf8982b3ad1"}

Method B:
Exception during scan: Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.
#0 /var/www/nextcloud/lib/private/Files/Stream/Encryption.php(517): OCA\Encryption\Crypto\Encryption->decrypt()
#1 /var/www/nextcloud/lib/private/Files/Stream/Encryption.php(316): OC\Files\Stream\Encryption->readCache()
#2 [internal function]: OC\Files\Stream\Encryption->stream_read()
#3 /var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php(55): fread()
#4 /var/www/nextcloud/3rdparty/icewind/streams/src/CallbackWrapper.php(96): Icewind\Streams\Wrapper->stream_read()
#5 [internal function]: Icewind\Streams\CallbackWrapper->stream_read()
#6 /var/www/nextcloud/lib/private/Preview/TXT.php(65): stream_get_contents()
#7 /var/www/nextcloud/lib/private/Preview/GeneratorHelper.php(64): OC\Preview\TXT->getThumbnail()
#8 /var/www/nextcloud/lib/private/Preview/Generator.php(361): OC\Preview\GeneratorHelper->getThumbnail()
#9 /var/www/nextcloud/lib/private/Preview/Generator.php(337): OC\Preview\Generator->generateProviderPreview()
#10 /var/www/nextcloud/lib/private/Preview/Generator.php(143): OC\Preview\Generator->getMaxPreview()
#11 /var/www/nextcloud/lib/private/Preview/Generator.php(110): OC\Preview\Generator->generatePreviews()
#12 /var/www/nextcloud/lib/private/PreviewManager.php(190): OC\Preview\Generator->getPreview()
#13 /var/www/nextcloud/lib/private/Blurhash/Listener/GenerateBlurhashMetadata.php(90): OC\PreviewManager->getPreview()
#14 /var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php(86): OC\Blurhash\Listener\GenerateBlurhashMetadata->handle()
#15 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(230): OC\EventDispatcher\ServiceEventListener->__invoke()
#16 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(59): Symfony\Component\EventDispatcher\EventDispatcher->callListeners()
#17 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(86): Symfony\Component\EventDispatcher\EventDispatcher->dispatch()
#18 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(98): OC\EventDispatcher\EventDispatcher->dispatch()
#19 /var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php(114): OC\EventDispatcher\EventDispatcher->dispatchTyped()
#20 /var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php(121): OC\FilesMetadata\FilesMetadataManager->refreshMetadata()
#21 /var/www/nextcloud/apps/files/lib/Command/Scan.php(143): OC\FilesMetadata\FilesMetadataManager->refreshMetadata()
#22 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}()
#23 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array()
#24 /var/www/nextcloud/lib/private/Hooks/PublicEmitter.php(40): OC\Hooks\BasicEmitter->emit()
#25 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(134): OC\Hooks\PublicEmitter->emit()
#26 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}()
#27 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(105): call_user_func_array()
#28 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(177): OC\Hooks\BasicEmitter->emit()
#29 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(529): OC\Files\Cache\Scanner->scanFile()
#30 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(453): OC\Files\Cache\Scanner->handleChildren()
#31 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(359): OC\Files\Cache\Scanner->scanChildren()
#32 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(278): OC\Files\Cache\Scanner->scan()
#33 /var/www/nextcloud/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan()
#34 /var/www/nextcloud/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles()
#35 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()
#36 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#37 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#38 /var/www/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#39 /var/www/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#40 /var/www/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()
#41 /var/www/nextcloud/console.php(102): OC\Console\Application->run()
#42 /var/www/nextcloud/occ(11): require_once('...')
#43 {main}

Additional info

The same underlying problem is maybe also the reason for a bug when using the fulltextsearch with user-specific encryption.

One can partially workaround this bug by temporarily disabling the preview functionality by adding

'enable_previews' => false,

in the config file. This helps for all cases where Preview is involved. By I still get the same error when uploading new files through the desktop client and after first time the cron job runs (and every succeeding cron job run):

{"reqId":"EArIZofvS1EatxYt5dkh","level":2,"time":"2024-08-16T20:15:01+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"issue while running UpdateSingleMetadata","userAgent":"--","version":"29.0.4.1","exception":{"Exception":"OC\\Encryption\\Exceptions\\DecryptionFailedException","Message":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","Code":0,"Trace":[{"file":"/var/www/nextcloud/lib/private/Files/Stream/Encryption.php","line":517,"function":"decrypt","class":"OCA\\Encryption\\Crypto\\Encryption","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Stream/Encryption.php","line":316,"function":"readCache","class":"OC\\Files\\Stream\\Encryption","type":"->"},{"function":"stream_read","class":"OC\\Files\\Stream\\Encryption","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":240,"function":"stream_get_contents"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":1171,"function":"file_get_contents","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/View.php","line":584,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Node/File.php","line":54,"function":"file_get_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nextcloud/lib/private/Blurhash/Listener/GenerateBlurhashMetadata.php","line":117,"function":"getContent","class":"OC\\Files\\Node\\File","type":"->"},{"file":"/var/www/nextcloud/lib/private/Blurhash/Listener/GenerateBlurhashMetadata.php","line":96,"function":"resizedImageFromFile","class":"OC\\Blurhash\\Listener\\GenerateBlurhashMetadata","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":86,"function":"handle","class":"OC\\Blurhash\\Listener\\GenerateBlurhashMetadata","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":230,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":59,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":86,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":98,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php","line":114,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/FilesMetadata/Job/UpdateSingleMetadata.php","line":60,"function":"refreshMetadata","class":"OC\\FilesMetadata\\FilesMetadataManager","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/Job.php","line":80,"function":"run","class":"OC\\FilesMetadata\\Job\\UpdateSingleMetadata","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":61,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/lib/public/BackgroundJob/QueuedJob.php","line":47,"function":"start","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":177,"function":"execute","class":"OCP\\BackgroundJob\\QueuedJob","type":"->"}],"File":"/var/www/nextcloud/apps/encryption/lib/Crypto/Encryption.php","Line":340,"Hint":"Cannot decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","message":"issue while running UpdateSingleMetadata","exception":[],"userId":"<redacted>","fileId":<redacted>,"CustomMessage":"issue while running UpdateSingleMetadata"},"id":"66bfb345c8e80"}

As can be seen, this time not getPreview tries to access the encrypted file but now it is resizedImageFromFile. But it is always triggered for the blurhash generation.

In the end, disabling Preview still fills the log.

Start with lib/private/Blurhash/Listener/GenerateBlurhashMetadata.php and trace its getPreview call through lib/private/PreviewManager.php and the encryption stream files named in the stack trace. Reproduce with occ files:scan --generate-metadata or the cron job using user-key encryption; done means no DecryptionFailedException, the background job completes, and metadata generation succeeds.