[Bug]: Server does not utilize system CA Certificates through specified HTTP Proxy

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Nextcloud outbound HTTP communications through forward HTTP proxy specified in config.php does not utilize the system ca store (on Debian: /etc/ssl/certs/ca-certificates.crt) even when CA file is explicitly specified in both FPM and CLI php.ini configuration files. (cURL error 35 - unknown CA)

Steps to reproduce

1. Specify HTTP proxy in config.php (ex. 'proxy' => '172.20.20.10:3142',)
2. Ensure php.ini in both FPM and CLI have CA crt file specified (ex. openssl.cafile=/etc/ssl/certs/ca-certificates.crt)
3. Test nextcloud's outbound connection by browsing to the app store in your web browser.
4. See error (in nextcloud logs and squid logs)

Expected behavior

Nextcloud, when using an HTTP proxy, should trust all CA certificates in system store.

Installation method

N/A

Nextcloud Server version

28

Operating system

Debian 12

PHP engine version

PHP 8.3

Web server

NGINX

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Unk/N/A

Are you using the Nextcloud Server Encryption module?

No

What user-backends are you using?

• [ x] Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "proxy": "apt.example.com:3142",
        "proxyexclude": [
            "nextcloud.example.com"
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "172.20.20.5",
            "nextcloud.example.com",
            "172.20.20.6"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.3.4",
        "overwrite.cli.url": "http:\/\/nextcloud.example.com\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "maintenance": false,
        "maintenance_window_start": 1,
        "theme": "",
        "loglevel": 2,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0,
            "timeout": 1.5
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "default_phone_region": "US",
        "allow_local_remote_servers": true,
        "mail_smtpmode": "sendmail",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "app_install_overwrite": [
            "breezedark"
        ],
        "app.mail.verify-tls-peer": false,
        "memories.exiftool_no_local": true,
        "memories.vod.path": "\/var\/www\/nextcloud\/apps\/memories\/bin-ext\/go-vod-amd64"
    }
}

List of activated Apps

Enabled:
  - admin_audit: 1.19.0
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.12
  - checksum: 1.2.4
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - collectives: 2.12.0
  - comments: 1.19.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - external: 5.4.0
  - federatedfilesharing: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.7.5
  - maps: 1.4.0
  - notes: 4.10.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.3
  - riotchat: 0.17.6
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0
  - tables: 0.7.4
  - text: 3.10.1
  - theming: 2.4.0
  - theming_customcss: 1.16.0
  - twofactor_backupcodes: 1.18.0
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - activity: 2.21.1 (installed 2.13.4)
  - encryption: 2.17.0
  - federation: 1.19.0 (installed 1.10.1)
  - files_archive: 1.2.3 (installed 1.2.3)
  - files_external: 1.21.0
  - firstrunwizard: 2.18.0 (installed 2.9.0)
  - nextcloud_announcements: 1.18.0 (installed 1.9.0)
  - recommendations: 2.1.0 (installed 0.8.0)
  - sharebymail: 1.19.0 (installed 1.10.0)
  - support: 1.12.0 (installed 1.3.0)
  - survey_client: 1.17.0 (installed 1.8.0)
  - twofactor_totp: 11.0.0-dev
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs - Limited (Github 65K character limit)

{"reqId":"LhQTAEFZsygTyEd2JOSG","level":2,"time":"2024-07-18T19:26:46+00:00","remoteAddr":"10.10.0.2","user":"bigdog","app":"appstoreFetcher","method":"GET","url":"/settings/apps","message":"Could not connect to appstore: cURL error 35: OpenSSL/3.0.13: error:8000000D:system library::Permission denied (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","version":"29.0.3.4","data":{"app":"appstoreFetcher"}}
{"reqId":"aptq5AybKgsT7H77SUWE","level":3,"time":"2024-07-18T19:32:28+00:00","remoteAddr":"10.10.0.2","user":"bigdog","app":"internet_connection_check","method":"GET","url":"/settings/ajax/checksetup","message":"Cannot connect to: www.nextcloud.com","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0","version":"29.0.3.4","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 35: OpenSSL/3.0.13: error:8000000D:system library::Permission denied (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://nextcloud.com/","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":142,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":123,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":107,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":73,"function":"checkRedirect","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/FulfilledPromise.php","line":41,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\RedirectMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/TaskQueue.php","line":48,"function":"GuzzleHttp\\Promise\\{closure}","class":"GuzzleHttp\\Promise\\FulfilledPromise","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":248,"function":"run","class":"GuzzleHttp\\Promise\\TaskQueue","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":224,"function":"invokeWaitFn","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":269,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":226,"function":"invokeWaitList","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/promises/src/Promise.php","line":62,"function":"waitIfPending","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":189,"function":"wait","class":"GuzzleHttp\\Promise\\Promise","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":230,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/SetupChecks/InternetConnectivity.php","line":85,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/SetupChecks/InternetConnectivity.php","line":68,"function":"isSiteReachable","class":"OCA\\Settings\\SetupChecks\\InternetConnectivity","type":"->"},{"file":"/var/www/nextcloud/lib/private/SetupCheck/SetupCheckManager.php","line":51,"function":"run","class":"OCA\\Settings\\SetupChecks\\InternetConnectivity","type":"->"},{"file":"/var/www/nextcloud/apps/settings/lib/Controller/CheckSetupController.php","line":179,"function":"runAll","class":"OC\\SetupCheck\\SetupCheckManager","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":232,"function":"check","class":"OCA\\Settings\\Controller\\CheckSetupController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":138,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":338,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/lib/base.php","line":1050,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":49,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"message":"Cannot connect to: www.nextcloud.com","exception":{},"CustomMessage":"Cannot connect to: www.nextcloud.com"}}

Additional info

No response

Start with lib/private/Http/Client/Client.php and the Guzzle call shown in the trace, then compare how the proxy is used during apps/settings/lib/SetupChecks/InternetConnectivity.php. Reproduce the cURL error with the stated proxy and system CA configuration; done means outbound HTTPS requests through the configured proxy trust the system CA store.