[Bug]: Text app saving/versioning is not consistent

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I am observing data in my S3 bucket (AWS) inconsistent with content on my Nextcloud server. See repro steps for more details, I did came across this issue https://github.com/nextcloud/server/issues/41917 which may or may not be related, my symptoms are slightly different and this is not a new issue - I observed it for two months, I was waiting for https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/326 to be fixed before reporting this on the off chance it was related.

Steps to reproduce

1. Mount S3 external folder.
2. Create a new blank file test.md via NextCloud UI, click save, exit the editor.
3. Look into Details -> Versions, there will be Initial version and Current version, both 0 B.
4. Look on S3 - you will also see two distinct versions, the first will always be Standard storage class and second (current) will be Intelligent-Tiering - not sure if this is AWS or NextCloud doing, I would expect them both be Intelligent-Tiering since my bucket is configured for it. This is a first sign of trouble, that there are two versions from get-go and that one version is a wrong storage class.
5. Open test.md in UI, add first line content, save the file, exit the editor.
6. Observe in Details -> Version that there is now a third version Current version correctly opening. The first two empty versions do not open, they are presenting a loading spinner forever and nothing pops up in the logs.
7. Observe on the S3 bucket that you have a new third (current) version with a correct content of first line.
8. Open test.md in UI, add second line content, save the file, exit the editor.
9. Observe in Details -> Version that there is now a new version, with correct new content.
10. Observe on the S3 bucket that you have a new (current) version with a correct content.
11. Open test.md in UI, add third line content, save the file, exit the editor.
12. Observe in Details -> Version that there is now a new version, with correct new content.
13. Observe on the S3 bucket that you still have old version as current with only two lines, and third line nowhere to be seen. I left it like that overnight thinking it may eventually sync, but it didn't. No errors in logs.
14. Adding more changes will trigger sync again. It is some kind of a race condition, sometimes I get inconsistent data on odd lines, sometimes - on even, sometimes it just works. Sometimes it is missing to create local versions too.

Additional details

I did not notice causality between how fast I am making changes and inconsistencies, it may occur on singular isolated edits. Although I did notice correlation, it is occurring more often on fast edits, which is a typical use case for a Markdown KB (add a thought, save, edit, save etc - and you never know if your latest edit actually made it to S3).

You may notice under disabled apps I have files_versions_s3 - I tried to install it to see if it makes any difference, it didn't. It was having this behavior before I ever installed it.

Expected behavior

Expected to see clear linear version history both in NextCloud and S3, and most importantly - expecting to always see latest and consistent content on S3. In my home use scenario, S3 should be the most reliable storage for most important documents and KB, because my home lab hardware may always be stolen or destroyed, and traditional scheduled backups are never latest. This bug is preventing me from fully trusting NextCloud over Google Drive. I may never be sure that my off-site data is consistent.

Installation method

Official All-in-One appliance

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Other

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

# docker exec --user www-data nextcloud-aio-nextcloud php occ config:list system
{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "27.1.4.1",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "loglevel": "2",
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita",
            "0": "OC\\Preview\\Imaginary"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "updatedirectory": "\/nc-updater",
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "allow_local_remote_servers": true,
        "preview_imaginary_url": "http:\/\/nextcloud-aio-imaginary:9000",
        "default_phone_region": "US",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587 ",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "data-fingerprint": "d810320451ca5c99313f7c2bbc7e472f",
        "appsallowlist": []
    }
}

List of activated Apps

# docker exec --user www-data nextcloud-aio-nextcloud php occ app:list
Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - announcementcenter: 6.7.0
  - app_api: 1.3.0
  - calendar: 4.5.3
  - cfg_share_links: 4.2.0
  - checksum: 1.2.2
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.4.2
  - dashboard: 7.7.0
  - dav: 1.27.0
  - deck: 1.11.2
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_antivirus: 5.4.0
  - files_archive: 1.1.3
  - files_external: 1.19.0
  - files_fulltextsearch: 27.0.1
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - firstrunwizard: 2.16.0
  - fulltextsearch: 27.0.2
  - fulltextsearch_elasticsearch: 27.0.5
  - groupfolders: 15.3.1
  - imageconverter: 1.3.5
  - integration_giphy: 1.0.8
  - integration_google: 2.1.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - metadata: 0.19.0
  - nextcloud-aio: 0.4.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - notify_push: 0.6.5
  - oauth2: 1.15.1
  - oidc: 0.7.1
  - password_policy: 1.17.0
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - richdocuments: 8.2.3
  - secrets: 1.5.2
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - side_menu: 3.11.2
  - socialsharing_email: 3.0.0
  - spreed: 17.1.3
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_nextcloud_notification: 3.8.0
  - twofactor_totp: 9.0.0
  - twofactor_webauthn: 1.3.2
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - bruteforcesettings: 2.7.0
  - contactsinteraction: 1.8.0 (installed 1.8.0)
  - encryption: 2.15.0
  - files_versions_s3: 0.2.1 (installed 0.2.1)
  - suspicious_login: 5.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

No source file or test is named in the issue. First reproduce the save/versioning sequence with a Markdown file in an S3 external folder, then inspect the Text app's save flow and its interaction with Nextcloud versions and external storage. Done means repeated saves produce a linear, latest-content version history consistently in both Nextcloud and S3.