nextcloud / nextcloud/server

Files are not deleted from S3 (primary)

Open
#20,333 43 comments 45 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage 18-feedback 25-feedback bug feature: object storage feature: trashbin needs review
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

How to use GitHub
  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
Steps to reproduce
  1. Set S3 as primary storage
  2. Upload, say, 2000 files into a folder (JPGs here)
  3. Delete that folder, and try to empty trashbin
Expected behaviour

Trashbin should be empty correctly

Actual behaviour

After some time, an error appears (Error while empty trash). Reloading the page shows no more files either in the Files or Trashbin.

image

But files still on the Object Storage, here OBJECTS and SIZE:

image

Before these test operations (upload, delete...)

image

Following commands had been executed (after):

sudo -u testing php occ files:scan test

Starting scan for user 1 out of 1 (test)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0       | 0     | 00:00:00     |
+---------+-------+--------------+

sudo -u testing php occ files:cleanup

0 orphaned file cache entries deleted

sudo -u www-data php occ trashbin:cleanup --all-users

Remove deleted files for all users
Remove deleted files for users on backend Database
   test

One user has reported that interface show he is "using" 1,9Gb of storage, but he has NO FILES or FOLDERS at all, either in FILES or TRASHBIN in a production instance.

Server configuration

Operating system: Ubuntu 18.04

Web server: Nginx 17

Database: MariaDB 10.4

PHP version: 7.3

Nextcloud version: (see Nextcloud admin page) 18.0.3

Updated from an older Nextcloud/ownCloud or fresh install: Fresh install

Where did you install Nextcloud from: Official sources

Signing status:

Signing status 
No errors have been found.

List of activated apps:

App list 
Enabled:
  - accessibility: 1.4.0
  - admin_audit: 1.8.0
  - announcementcenter: 3.7.0
  - apporder: 0.9.0
  - cloud_federation_api: 1.1.0
  - dav: 1.14.0
  - external: 3.5.0
  - federatedfilesharing: 1.8.0
  - files: 1.13.1
  - files_accesscontrol: 1.8.1
  - files_automatedtagging: 1.8.2
  - files_pdfviewer: 1.7.0
  - files_rightclick: 0.15.2
  - files_sharing: 1.10.1
  - files_trashbin: 1.8.0
  - files_versions: 1.11.0
  - files_videoplayer: 1.7.0
  - groupfolders: 6.0.3
  - impersonate: 1.5.0
  - logreader: 2.3.0
  - lookup_server_connector: 1.6.0
  - notifications: 2.6.0
  - oauth2: 1.6.0
  - password_policy: 1.8.0
  - privacy: 1.2.0
  - provisioning_api: 1.8.0
  - settings: 1.0.0
  - sharebymail: 1.8.0
  - theming: 1.9.0
  - theming_customcss: 1.5.0
  - twofactor_backupcodes: 1.7.0
  - viewer: 1.2.0
  - workflow_script: 1.3.1
  - workflowengine: 2.0.0

Nextcloud configuration:

Config report 
{
    "system": {
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "testing.example.com",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "10.1.0.2",
                "port": 8080,
                "use_ssl": false,
                "region": "fr-par",
                "use_path_style": true
            }
        },
        "log_type": "file",
        "logfile": "\/var\/log\/nextcloud\/testing.example.com-nextcloud.log",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "testing.example.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "18.0.3.0",
        "overwrite.cli.url": "https:\/\/testing.example.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "3306",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "dbdriveroptions": {
            "1009": "\/etc\/ssl\/mysql\/ca-cert.pem",
            "1008": "\/etc\/ssl\/mysql\/client-cert.pem",
            "1007": "\/etc\/ssl\/mysql\/client-key.pem",
            "1014": false
        },
        "installed": true,
        "skeletondirectory": "",
        "default_language": "fr",
        "default_locale": "fr_FR",
        "activity_expire_days": 30,
        "auth.bruteforce.protection.enabled": false,
        "blacklisted_files": [
            ".htaccess",
            "Thumbs.db",
            "thumbs.db"
        ],
        "htaccess.RewriteBase": "\/",
        "integrity.check.disabled": false,
        "knowledgebaseenabled": false,
        "logtimezone": "Europe\/Paris",
        "maintenance": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "updatechecker": false,
        "appstoreenabled": false,
        "upgrade.disable-web": true,
        "filelocking.enabled": false,
        "overwriteprotocol": "https",
        "preview_max_scale_factor": 1,
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 2.5,
            "dbindex": 2,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "quota_include_external_storage": false,
        "theme": "",
        "trashbin_retention_obligation": "auto, 7",
        "updater.release.channel": "stable",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "testing.example.com",
        "preview_max_x": "1280",
        "preview_max_y": "800",
        "jpeg_quality": "70",
        "loglevel": 2,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap"
        ],
        "apps_paths": [
            {
                "path": "\/var\/www\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/custom",
                "url": "\/custom_apps",
                "writable": true
            }
        ]
    }
}

Logs are completely empty (we have just fired up a test instance, and test this use case).

Similar to https://github.com/nextcloud/server/issues/17744

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the S3 objectstore configuration and the reported reproduction: upload many files, delete the folder, then empty the trashbin. Compare the behavior of occ trashbin:cleanup --all-users, occ files:scan, and occ files:cleanup; done means the trashbin operation completes without error and the corresponding objects are removed from S3.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.