Error tying to register ISO/Template with DirectDownload over HTTPS with Let's Encrypt
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 19h
- Merged PRs (30d)
- 32
Description
### problem
When using the "Direct Download" feature for an ISO or Template (bypassing Secondary Storage), the Agent fails to verify Let's Encrypt certificates due to the absence of Let's Encrypt’s main CA certificate.
> Note: Let's Encrypt is widely used on the internet (>50% of all certificates).
ACS currently loads and trusts certificates exclusively from `/etc/cloudstack/agent/cloud.jks` and does not fall back to Java (`/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts`) or the system store (`/etc/ssl/certs/ca-certificates.crt`). Both of these contain the missing certificate (**ISRG Root X1**), which has been in use since 2015.
See: https://letsencrypt.org/certificates/
**ISRG Root X1** is the current root of the trust chain (valid until 2030), after which it will be replaced by **ISRG Root X2**.
**Recommendation:** Add a fallback to Java’s trust store to avoid maintaining an ever-changing list of certificates.
**Alternative:** As a short-term fix, include the missing CA certificate (https://letsencrypt.org/certs/isrgrootx1.pem) in `/etc/cloudstack/agent/cloud.jks` for the next release, while a more sustainable solution is developed.
### versions
We are running ACS 4.20.2 on Ubuntu 24.04. However, this issue likely affects all versions starting from 4.19, when the feature to bypass Secondary Storage was introduced.
Related issues and PRs:
- https://github.com/apache/cloudstack/issues/7929
- https://github.com/apache/cloudstack/pull/7693/changes
- https://github.com/apache/cloudstack/pull/7923/changes
- https://github.com/apache/cloudstack/pull/7932/changes
- https://github.com/apache/cloudstack/pull/11113/changes
### The steps to reproduce the bug
1. When registering an ISO or Template for Direct Download, use any HTTPS URL whose TLS certificate is issued by Let's Encrypt.
### What to do about it?
As a workaround, the following command can be run for each Zone to add the missing certificate. Note that this introduces additional manual steps for platform maintenance:
```
cmk upload templatedirectdownloadcertificate hypervisor="KVM" name="isrg-root-x1-2" certific
ate="$(curl -s https://letsencrypt.org/certs/isrgrootx1.pem)" zoneid="00000000-0000-0000-00000-000000000000"
```
Contributor guide
Research direction
Start with the Agent's certificate handling and `/etc/cloudstack/agent/cloud.jks`, then compare it with Java's `/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts` and the system store at `/etc/ssl/certs/ca-certificates.crt`. Reproduce direct ISO or Template download against an HTTPS URL using Let's Encrypt, and consider the related issues and PRs; done means the Agent verifies the certificate without a per-Zone manual workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100