Deliver up-to-date CA certificates to ExApps without rebuilding their images

Problem

An ExApp trusts whatever CA certificates were baked into its image on the day the image was built. Public CAs keep introducing new roots, and admins use private PKIs, so over time an ExApp image starts rejecting perfectly valid certificates of the Nextcloud instance it has to call back.

The typical symptom: the ExApp container starts, AppAPI reaches it (/heartbeat and /init answer 200), but the ExApp's own request to Nextcloud fails with CERTIFICATE_VERIFY_FAILED (unable to get local issuer certificate or self-signed certificate in certificate chain). The deploy then hangs at the init step. For the admin this is hard to read, because the instance certificate is valid in every browser.

"Test Deploy" is hit first, since it is the first thing an admin runs, and its image is only rebuilt manually. Rebuilding the image fixes it for a while (nextcloud/test-deploy#44), but that does not scale: every ExApp image has the same problem, and we cannot rebuild and re-release all of them each time a CA ships a new root.

What happens today

• Docker and HaRP (Docker) daemons: at deploy time AppAPI reads the Nextcloud CA bundle (ICertificateManager::getAbsoluteBundlePath(), i.e. the bundle shipped with the server plus everything imported with occ security:certificates:import), copies the certificates into the container and runs update-ca-certificates / update-ca-trust there (introduced for #411). This works, with limits:
  • it runs only on deploy and update, so certificates imported later do not reach ExApps that are already running;
  • it needs a supported OS, the update tool in the image, a writable root filesystem and root inside the container;
  • it only helps clients that read the OS trust store. Clients with their own bundle (certifi based ones, Node.js, Java) ignore it.
• Kubernetes daemon: AppAPI sends the same bundle to HaRP, but /k8s/exapp/install_certificates is a no-op and nothing is mounted into the ExApp pod. An ExApp on Kubernetes only ever has the CA store of its image, so neither newer public roots nor a private CA can be delivered at all. Exec-based installation would not be a good fit there anyway (read-only root filesystems, non-root policies).
• Manual daemons: nothing is delivered, which is fine, the developer owns that environment.

Goal

Collect ideas for a way to hand ExApps an up-to-date set of trusted CAs from the Nextcloud side, so that:

• a valid instance certificate works without rebuilding any ExApp image,
• certificates imported into Nextcloud reach ExApps on all deploy backends, including Kubernetes,
• it works in hardened environments (non-root, read-only root filesystem, no exec into the container).

Ideas to evaluate

1. Mount a bundle instead of installing into the image. Deliver the Nextcloud CA bundle as a read-only file: a ConfigMap or Secret created by HaRP on Kubernetes, a bind mount or docker cp of a single file on Docker. No exec, no package tooling, works with a read-only root filesystem. On Kubernetes this needs configmaps (or secrets) permissions for the HaRP service account.
2. A documented contract for where the bundle is. Set the conventional environment variables on every ExApp (SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, NODE_EXTRA_CA_CERTS) and possibly an AppAPI specific one, so SDKs can pick it up explicitly. nc_py_api already has NPA_NC_CERT for this, other languages would get the same rule in the ExApp development docs.
3. Refresh without redeploy. Re-deliver the bundle when it changes (certificate imported or removed, server update brings a newer bundle), for example on ExApp enable, through a background job, or with an occ command. With a mounted ConfigMap the update can propagate without touching the pod.
4. Let the admin reference an existing trust source. A daemon option that points to a ConfigMap or Secret already managed in the cluster (e.g. by trust-manager), for environments that manage trust centrally.
5. Keep images fresh anyway. Scheduled rebuilds of test-deploy (and a recommendation for ExApp authors) so base image updates are picked up. This only mitigates the public root case and does nothing for private CAs, so it complements the points above rather than replaces them.
6. Better diagnostics. When init fails because of TLS verification, surface that reason in the Test Deploy dialog and in the ExApp status instead of a generic init timeout, and make the Test Deploy log message point to certificates rather than suggesting plain HTTP.

Open questions

• Should the delivered bundle replace the image's CA store or be added to it?
• One mechanism for all backends (file mount plus environment variables), or keep the in-container installation on Docker and add the mount only for Kubernetes?
• How do ExApps whose HTTP client ignores both the OS store and the environment variables opt in?
• Where does the work split between app_api and HaRP, and does it need a new HaRP API version?

Start by tracing the existing CA delivery around ICertificateManager::getAbsoluteBundlePath() and the /k8s/exapp/install_certificates entry point, then compare Docker, Kubernetes and manual daemon behavior. Review the /heartbeat and /init flow, nc_py_api's NPA_NC_CERT contract, and the Test Deploy diagnostics. Done means a decided cross-backend design covering refresh, hardened environments, client compatibility and the work split with HaRP.