apache / apache/cloudstack

Error tying to register ISO/Template with DirectDownload over HTTPS with Let's Encrypt

Open
#12,878 2 comments 1 reaction 0 assignees View on GitHub
component:kvm component:templates no-issue-activity
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

When using the "Direct Download" feature for an ISO or Template (bypassing Secondary Storage), the Agent fails to verify Let's Encrypt certificates due to the absence of Let's Encrypt’s main CA certificate.

> Note: Let's Encrypt is widely used on the internet (>50% of all certificates).

ACS currently loads and trusts certificates exclusively from `/etc/cloudstack/agent/cloud.jks` and does not fall back to Java (`/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts`) or the system store (`/etc/ssl/certs/ca-certificates.crt`). Both of these contain the missing certificate (**ISRG Root X1**), which has been in use since 2015.

See: https://letsencrypt.org/certificates/

Image

**ISRG Root X1** is the current root of the trust chain (valid until 2030), after which it will be replaced by **ISRG Root X2**.

**Recommendation:** Add a fallback to Java’s trust store to avoid maintaining an ever-changing list of certificates.

**Alternative:** As a short-term fix, include the missing CA certificate (https://letsencrypt.org/certs/isrgrootx1.pem) in `/etc/cloudstack/agent/cloud.jks` for the next release, while a more sustainable solution is developed.

### versions

We are running ACS 4.20.2 on Ubuntu 24.04. However, this issue likely affects all versions starting from 4.19, when the feature to bypass Secondary Storage was introduced.

Related issues and PRs:
- https://github.com/apache/cloudstack/issues/7929
- https://github.com/apache/cloudstack/pull/7693/changes
- https://github.com/apache/cloudstack/pull/7923/changes
- https://github.com/apache/cloudstack/pull/7932/changes
- https://github.com/apache/cloudstack/pull/11113/changes

### The steps to reproduce the bug

1. When registering an ISO or Template for Direct Download, use any HTTPS URL whose TLS certificate is issued by Let's Encrypt.

### What to do about it?

As a workaround, the following command can be run for each Zone to add the missing certificate. Note that this introduces additional manual steps for platform maintenance:

```
cmk upload templatedirectdownloadcertificate hypervisor="KVM" name="isrg-root-x1-2" certific
ate="$(curl -s https://letsencrypt.org/certs/isrgrootx1.pem)" zoneid="00000000-0000-0000-00000-000000000000"
```

Contributor guide

Open the contributing guide

Research direction

Start with the Agent's certificate handling and `/etc/cloudstack/agent/cloud.jks`, then compare it with Java's `/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts` and the system store at `/etc/ssl/certs/ca-certificates.crt`. Reproduce direct ISO or Template download against an HTTPS URL using Let's Encrypt, and consider the related issues and PRs; done means the Agent verifies the certificate without a per-Zone manual workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.