[Bug]: File uploads intermittently fail with "Object not found after writing" due to object storage race condition

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I have encountered a bug where our Nextcloud server cannot manage to upload files, in a normal or team folder:

• File uploads fail with "Object not found after writing" error due to a race condition in object storage validation.

Steps to reproduce

1. Use Nextcloud with object storage backend (S3/Swift/etc.)
2. Upload files
3. Error occurs intermittently due to timing
4. The timing-dependent nature makes it difficult to reproduce consistently.

Expected behavior

Retry mechanism & File uploads should succeed

Nextcloud Server version

32

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 31 to 32)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwriteprotocol": "https",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "fneaurabuckets3",
                "region": "sbg",
                "hostname": "s3.sbg.io.cloud.ovh.net",
                "port": "443",
                "storageClass": "",
                "objectPrefix": "urn:oid:",
                "autocreate": true,
                "use_ssl": true,
                "use_path_style": true,
                "legacy_auth": false,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***"
            }
        },
        "upgrade.disable-web": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "intranet.next.fne-aura.org"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "32.0.0.13",
        "overwrite.cli.url": "https:\/\/intranet.next.fne-aura.org",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "default_phone_region": "FR",
        "enable_previews": false,
        "maintenance_window_start": 1,
        "default_language": "fr",
        "default_locale": "fr_FR",
        "default_timezone": "Europe\/Paris",
        "auto_logout": true,
        "knowledgebaseenabled": false,
        "skeletondirectory": "",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 0,
        "force_language": "fr",
        "force_locale": "fr_FR",
        "trashbin_retention_obligation": "auto, 90",
        "versions_retention_obligation": "disabled",
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "app_install_overwrite": [
            "libresign"
        ],
        "maintenance": false,
        "defaultapp": "dashboard,files,activity"
    }
}

List of activated Apps

Enabled:
  - activity: 5.0.0-dev.0
  - app_api: 32.0.0
  - bruteforcesettings: 5.0.0-dev.0
  - calendar: 6.0.1
  - circles: 32.0.0
  - cloud_federation_api: 1.16.0
  - comments: 1.22.0
  - contacts: 8.0.3
  - contactsinteraction: 1.13.1
  - dashboard: 7.12.0
  - dav: 1.34.2
  - deck: 1.16.0
  - federatedfilesharing: 1.22.0
  - federation: 1.22.0
  - files: 2.4.0
  - files_downloadlimit: 5.0.0-dev.0
  - files_pdfviewer: 5.0.0-dev.0
  - files_reminders: 1.5.0
  - files_sharing: 1.24.0
  - files_trashbin: 1.22.0
  - files_versions_s3: 1.3.0
  - firstrunwizard: 5.0.0-dev.0
  - forms: 5.2.1
  - impersonate: 3.0.0
  - libresign: 11.4.1
  - logreader: 5.0.0-dev.0
  - lookup_server_connector: 1.20.0
  - mail: 5.5.7
  - nextcloud_announcements: 4.0.0-dev.0
  - notes: 4.12.3
  - notifications: 5.0.0-dev.0
  - notify_push: 1.2.0
  - oauth2: 1.20.0
  - password_policy: 4.0.0-dev.0
  - polls: 8.5.0
  - privacy: 4.0.0-dev.0
  - profile: 1.1.0
  - provisioning_api: 1.22.0
  - recommendations: 5.0.0-dev.0
  - related_resources: 3.0.0-dev.0
  - richdocuments: 9.0.0
  - serverinfo: 4.0.0-dev.0
  - settings: 1.15.1
  - sharebymail: 1.22.0
  - spreed: 22.0.0
  - survey_client: 4.0.0-dev.0
  - systemtags: 1.22.0
  - text: 6.0.0-dev.0
  - theming: 2.7.0
  - twofactor_backupcodes: 1.21.0
  - updatenotification: 1.22.0
  - user_status: 1.12.0
  - viewer: 5.0.0-dev.0
  - weather_status: 1.12.0
  - webhook_listeners: 1.3.0
  - workflowengine: 2.14.0
Disabled:
  - admin_audit: 1.22.0 (installed 1.21.0)
  - collectives: 3.2.0 (installed 3.2.0)
  - encryption: 2.20.0 (installed 2.19.0)
  - files_external: 1.24.0 (installed 1.24.0)
  - files_versions: 1.25.0 (installed 1.24.0)
  - photos: 5.0.0-dev.1 (installed 2.5.0)
  - support: 4.0.0-dev.0 (installed 3.0.0)
  - suspicious_login: 10.0.0-dev.0
  - twofactor_nextcloud_notification: 6.0.0-dev.0
  - twofactor_totp: 14.0.0
  - user_ldap: 1.23.0 (installed 1.22.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Additional info

We use S3 bucket provided by OVH. Maybe not fast as aws s3.

In the upload code, objectExists() verification happens immediately after writing, but object storage has latency.

Maybe add retry mechanism or make validation optional for object storage backends.

see ->
https://github.com/nextcloud/server/blob/8d2a95678fd1db0dda0567efadb2d3d667aaa689/lib/private/Files/ObjectStore/ObjectStoreStorage.php#L556C3-L567C4

I've temporarily resolved this issue by removing/disabling the objectExists() validation check:

Start in lib/private/Files/ObjectStore/ObjectStoreStorage.php around lines 556-567, where objectExists() is checked after writing, and trace the upload path for object storage backends. Reproduce the intermittent failure with the reported S3/OVH setup and determine how validation should handle delayed visibility; done means uploads no longer fail incorrectly while object verification remains appropriately covered.