Large data files left in S3 object storage after bad uploads
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
How to use GitHub
- Please use the 👍 reaction to show that you are affected by the same issue.
- Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
- Subscribe to receive notifications on status change and new comments.
Steps to reproduce
- Send files via desktop client or web
- Abort sending by internet connection failure (or wait for desktop clients error timeout/mismatch byte count)
- Check S3 container content for 500 MB files with binary countent of no sense
Expected behaviour
Server should notice that upload is aborted and should delete this badly uploaded files from object storage.
Actual behaviour
Server leaves objects on object storage (OVH S3) in 500MB files. Cannot clear them using occ files:scan --all and occ files:cleanup.
Server configuration
Operating system: Ubuntu 20.04 LTS
Web server: Apache/2.4.41
Database: MySQL 10.3.31
PHP version: 7.4.3
Nextcloud version: 22.2.0.2
Updated from an older Nextcloud/ownCloud or fresh install: fresh install
Where did you install Nextcloud from: zip file
Signing status:
Signing status
No errors have been found.
List of activated apps:
App list
If you have access to your command line run e.g.:
Enabled:
- accessibility: 1.8.0
- activity: 2.15.0
- apporder: 0.13.0
- bruteforcesettings: 2.2.0
- circles: 22.1.1
- cloud_federation_api: 1.5.0
- comments: 1.12.0
- contacts: 4.0.6
- contactsinteraction: 1.3.0
- dashboard: 7.2.0
- dav: 1.19.0
- deck: 1.5.5
- federatedfilesharing: 1.12.0
- federation: 1.12.0
- files: 1.17.0
- files_external: 1.13.0
- files_pdfviewer: 2.3.0
- files_rightclick: 1.1.0
- files_sharing: 1.14.0
- files_trashbin: 1.12.0
- files_versions: 1.15.0
- files_videoplayer: 1.11.0
- firstrunwizard: 2.11.0
- groupfolders: 10.0.0
- logreader: 2.7.0
- lookup_server_connector: 1.10.0
- nextcloud_announcements: 1.11.0
- notes: 4.2.0
- notifications: 2.10.1
- oauth2: 1.10.0
- password_policy: 1.12.0
- privacy: 1.6.0
- provisioning_api: 1.12.0
- quota_warning: 1.11.0
- recommendations: 1.1.0
- serverinfo: 1.12.0
- settings: 1.4.0
- sharebymail: 1.12.0
- support: 1.5.0
- survey_client: 1.10.0
- systemtags: 1.12.0
- tasks: 0.14.2
- text: 3.3.0
- theming: 1.13.0
- twofactor_backupcodes: 1.11.0
- updatenotification: 1.12.0
- user_status: 1.2.0
- viewer: 1.6.0
- weather_status: 1.2.0
- workflowengine: 2.4.0
Disabled:
- admin_audit
- encryption
- photos
- user_ldap
Nextcloud configuration:
Config report
{
"system": {
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"***REMOVED SENSITIVE VALUE***"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"objectstore": {
"class": "\\OC\\Files\\ObjectStore\\S3",
"arguments": {
"bucket": "nextcloud",
"autocreate": true,
"key": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"hostname": "storage.waw.cloud.ovh.net",
"port": 443,
"region": "waw",
"use_ssl": true,
"use_path_style": true
}
},
"dbtype": "mysql",
"version": "22.2.0.2",
"overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"default_phone_region": "PL",
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_smtpmode": "smtp",
"mail_sendmailmode": "smtp",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtpauthtype": "LOGIN",
"mail_smtpauth": 1,
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "587",
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"has_rebuilt_cache": true
}
}
Are you using external storage, if yes which one: S3 Object Storage as default nextcloud data storage
Are you using encryption: no
Are you using an external user-backend, if yes which one: Not sure, probably no
Client configuration
Browser: Chromium-based 95.0.1020.53
Operating system: Windows 10, Windows 11, GNU/Linux
Desktop client version: 3.3.6 (Windows)
Logs
Web server error log
Server error log
[Mon Nov 22 08:56:47.033194 2021] [access_compat:error] [pid 424412] [client 209.141.34.220:46930] AH01797: client denied by server configuration: /var/www/html/config/getuser
[Mon Nov 22 13:53:42.980006 2021] [access_compat:error] [pid 429447] [client 209.141.34.220:49522] AH01797: client denied by server configuration: /var/www/html/config/getuser
[Mon Nov 22 16:55:45.673139 2021] [php7:error] [pid 434037] [client 213.231.8.6:56997] script '/var/www/html/wp-login.php' not found or unable to stat, referer: http://***PRIVATE HOSTNAME***/wp-login.php
Nextcloud log (data/nextcloud.log)
Nextcloud log
Not sure how to get this from S3
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the server's S3 object-storage upload and cleanup paths, using the reported abort scenario as the reproduction case. The issue also mentions occ files:scan --all and occ files:cleanup, which can help verify current cleanup behavior. Done means aborted or failed uploads no longer leave 500 MB objects in S3.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100