[Bug]: OCS API returns XML output on invalid auth even if requesting JSON responses

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

When using incorrect credentials and specifying Accept: application/json by request header or ?format=json in url arguments the authentication error will be given back to you in XML format rather than json.

Steps to reproduce

Send a request with invalid credentials:

$ curl -u invaliduser:invalidpass -H 'OCS-APIRequest: true'  -H 'Accept: application/json' http://localhost:8181/ocs/v1.php/cloud/users
<?xml version="1.0"?>
<ocs>
 <meta>
  <status>failure</status>
  <statuscode>997</statuscode>
  <message>Unauthorised</message>
 </meta>
 <data/>
</ocs>

Expected behavior

Error should be returned in JSON format.

This is only seen so far with invalid credentials. Requests that are properly authenticated but improperly formed in some other way will properly return errors in JSON format.

Installation method

Manual installation

Operating system

Debian/Ubuntu

PHP engine version

PHP 7.4

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

No response

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

`
{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "[REDACTED]",
            "[REDACTED]"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "23.0.6.1",
        "overwrite.cli.url": "[REDACTED]",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "filelocking.enabled": true,
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "has_rebuilt_cache": true,
        "maintenance": false,
        "loglevel": 0,
        "theme": "",
        "updater.release.channel": "stable",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "default_phone_region": "US",
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory"
    }
}

### List of activated Apps

```shell
Enabled:                      
  - accessibility: 1.9.0
  - activity: 2.15.0  
  - admin_audit: 1.13.0   
  - apporder: 0.15.0
  - breezedark: 23.2.2   
  - bruteforcesettings: 2.4.0
  - calendar: 3.4.2          
  - camerarawpreviews: 0.7.16
  - circles: 23.1.2
  - cloud_federation_api: 1.6.0
  - comments: 1.13.0
  - contacts: 4.1.1   
  - contactsinteraction: 1.4.0    
  - cookbook: 0.9.13      
  - dav: 1.21.0           
  - deck: 1.6.3              
  - federatedfilesharing: 1.13.0
  - files: 1.18.0
  - files_markdown: 2.3.6 
  - files_mindmap: 0.0.26
  - files_pdfviewer: 2.4.0
  - files_photospheres: 1.23.1
  - files_rightclick: 1.2.0
  - files_sharing: 1.15.0 
  - files_trashbin: 1.13.0 
  - files_versions: 1.16.0
  - files_videoplayer: 1.12.0
  - forms: 2.5.1
  - groupfolders: 11.1.5
  - logreader: 2.8.0
  - lookup_server_connector: 1.11.0
  - maps: 0.1.10
  - music: 1.5.2
  - nextcloud_announcements: 1.12.0
  - notes: 4.3.1
  - notifications: 2.11.1
  - oauth2: 1.11.0
  - ocdownloader: 1.8.0
  - password_policy: 1.13.0
  - passwords: 2022.6.10
  - phonetrack: 0.7.0
  - photos: 1.5.0
  - polls: 3.7.0
  - previewgenerator: 5.0.0
  - privacy: 1.7.0
  - provisioning_api: 1.13.0
  - rainloop: 7.2.5
  - serverinfo: 1.13.0
  - settings: 1.5.0
  - sharebymail: 1.13.0
  - socialsharing_email: 2.5.0
  - spreed: 13.0.7
  - systemtags: 1.13.0
  - tasks: 0.14.4
  - text: 3.4.1
  - theming: 1.14.0
  - twofactor_backupcodes: 1.12.0
  - unsplash: 1.2.5
  - updatenotification: 1.13.0
  - user_ldap: 1.13.1
  - user_status: 1.3.1
  - video_converter: 1.0.5
  - viewer: 1.7.0
  - workflowengine: 2.5.0
Disabled:
  - announcementcenter: 6.3.1
  - dashboard: 7.3.0
  - encryption
  - epubreader: 1.4.7
  - external: 3.10.2
  - federation: 1.13.0
  - files_automatedtagging: 1.13.0
  - files_external: 1.15.0
  - firstrunwizard: 2.12.0
  - integration_github: 1.0.2
  - integration_reddit: 1.0.4
  - news: 18.1.0
  - recommendations: 1.2.0
  - richdocuments: 5.0.6
  - support: 1.6.0
  - survey_client: 1.11.0
  - weather_status: 1.3.0
  - workflow_script: 1.8.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

Start by reproducing the request against the OCS endpoint shown with invalid credentials, Accept: application/json, and ?format=json. Trace the authentication-error response path and its format negotiation; done means invalid-auth responses honor the requested JSON format while preserving the existing XML behavior when JSON is not requested.