KVM agent PostCertificateRenewalTask fails with IllegalStateException during certificate provisioning
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 19h
- Merged PRs (30d)
- 32
Description
## Description
When `provisionCertificate` is called on a KVM host, the agent's `PostCertificateRenewalTask` consistently fails with `java.lang.IllegalStateException: Shutdown in progress`. This causes the host to permanently report `secured=false` in `StartupRoutingCommand.hostDetails`, showing as "Unsecure" in the UI despite having valid TLS certificates and a working SSL connection.
## Steps to Reproduce
1. Add a new KVM host to CloudStack 4.22 with `ca.plugin.root.auth.strictness=true`
2. Run `provisionCertificate hostid=`
3. The API returns `{"success": true}` — keystore, cert, CA cert, and key are all created correctly
4. The `PostCertificateRenewalTask` attempts to restart libvirtd and reconnect, but the cert provisioning triggers an agent restart
5. During the agent shutdown, `Runtime.removeShutdownHook()` throws `IllegalStateException: Shutdown in progress`
6. The agent never sets `secured=true` — the host permanently shows "Unsecure"
## Error Log (agent.log)
```
INFO [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Restarting libvirt after certificate provisioning/renewal
WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Execution of process for command [sudo service libvirtd restart ] failed.
WARN [cloud.agent.Agent] Failed to execute post certificate renewal command: java.lang.IllegalStateException: Shutdown in progress
at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
at com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1377)
```
## Root Cause
In `Agent.java:1377`, the `PostCertificateRenewalTask` calls `Runtime.getRuntime().removeShutdownHook()` during JVM shutdown, which is not allowed per Java spec. The certificate provisioning triggers an agent reconnect/restart, creating a race condition where the PostCertificateRenewal task runs during the shutdown window.
## Impact
- Host permanently shows "Unsecure" in the UI despite valid TLS
- The actual SSL connection works correctly (keystore loads, handshake succeeds)
- The `secured` flag in `host_details` DB table is overwritten to `false` on every agent reconnect
- Manual DB updates are overwritten by the agent's `StartupRoutingCommand`
- Reproduced consistently on multiple `provisionCertificate` attempts
## Suggested Fix
The `PostCertificateRenewalTask.runInContext()` should catch `IllegalStateException` from `removeShutdownHook()` and still proceed with setting the secured flag. Alternatively, check `Thread.currentThread().isInterrupted()` or use a guard flag before calling `removeShutdownHook()`.
```java
// In Agent.java PostCertificateRenewalTask.runInContext()
try {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
} catch (IllegalStateException e) {
// JVM is already shutting down, skip hook removal
LOG.debug("Skipping shutdown hook removal during shutdown", e);
}
```
## Environment
- CloudStack: 4.22.0.0
- OS: Ubuntu 22.04 (also reproduced on fresh provision)
- Java: OpenJDK 11.0.30 and 17.0.18
- KVM/libvirt: working correctly
- `ca.plugin.root.auth.strictness`: true
## Workaround
Manually update the DB: `UPDATE host_details SET value='true' WHERE host_id= AND name='secured';`
This is overwritten on next agent restart but the TLS connection is functionally secure regardless of the flag.
Contributor guide
Research direction
Start in Agent.java around line 1377 and inspect PostCertificateRenewalTask.runInContext(), especially the shutdown-hook removal during certificate provisioning. Reproduce with provisionCertificate on a KVM host using ca.plugin.root.auth.strictness=true and follow agent.log. Done means the shutdown race no longer leaves secured=false while preserving the successful certificate provisioning and reconnect flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cloud, infrastructure, security
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100