apache / apache/cloudstack

KVM agent PostCertificateRenewalTask fails with IllegalStateException during certificate provisioning

未关闭
#12,795 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
component:agent component:kvm Severity:Minor type:bug
主要语言
Java
星标
3.1k
派生
1.4k
平均合并
6 天 19 小时
30 天内合并 PR
32

描述

## Description

When `provisionCertificate` is called on a KVM host, the agent's `PostCertificateRenewalTask` consistently fails with `java.lang.IllegalStateException: Shutdown in progress`. This causes the host to permanently report `secured=false` in `StartupRoutingCommand.hostDetails`, showing as "Unsecure" in the UI despite having valid TLS certificates and a working SSL connection.

## Steps to Reproduce

1. Add a new KVM host to CloudStack 4.22 with `ca.plugin.root.auth.strictness=true`
2. Run `provisionCertificate hostid=`
3. The API returns `{"success": true}` — keystore, cert, CA cert, and key are all created correctly
4. The `PostCertificateRenewalTask` attempts to restart libvirtd and reconnect, but the cert provisioning triggers an agent restart
5. During the agent shutdown, `Runtime.removeShutdownHook()` throws `IllegalStateException: Shutdown in progress`
6. The agent never sets `secured=true` — the host permanently shows "Unsecure"

## Error Log (agent.log)

```
INFO [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Restarting libvirt after certificate provisioning/renewal
WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Execution of process for command [sudo service libvirtd restart ] failed.
WARN [cloud.agent.Agent] Failed to execute post certificate renewal command: java.lang.IllegalStateException: Shutdown in progress
at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
at com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1377)
```

## Root Cause

In `Agent.java:1377`, the `PostCertificateRenewalTask` calls `Runtime.getRuntime().removeShutdownHook()` during JVM shutdown, which is not allowed per Java spec. The certificate provisioning triggers an agent reconnect/restart, creating a race condition where the PostCertificateRenewal task runs during the shutdown window.

## Impact

- Host permanently shows "Unsecure" in the UI despite valid TLS
- The actual SSL connection works correctly (keystore loads, handshake succeeds)
- The `secured` flag in `host_details` DB table is overwritten to `false` on every agent reconnect
- Manual DB updates are overwritten by the agent's `StartupRoutingCommand`
- Reproduced consistently on multiple `provisionCertificate` attempts

## Suggested Fix

The `PostCertificateRenewalTask.runInContext()` should catch `IllegalStateException` from `removeShutdownHook()` and still proceed with setting the secured flag. Alternatively, check `Thread.currentThread().isInterrupted()` or use a guard flag before calling `removeShutdownHook()`.

```java
// In Agent.java PostCertificateRenewalTask.runInContext()
try {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
} catch (IllegalStateException e) {
// JVM is already shutting down, skip hook removal
LOG.debug("Skipping shutdown hook removal during shutdown", e);
}
```

## Environment

- CloudStack: 4.22.0.0
- OS: Ubuntu 22.04 (also reproduced on fresh provision)
- Java: OpenJDK 11.0.30 and 17.0.18
- KVM/libvirt: working correctly
- `ca.plugin.root.auth.strictness`: true

## Workaround

Manually update the DB: `UPDATE host_details SET value='true' WHERE host_id= AND name='secured';`
This is overwritten on next agent restart but the TLS connection is functionally secure regardless of the flag.

贡献指南

打开贡献指南

调研方向

从 Agent.java 第 1377 行附近开始,检查 PostCertificateRenewalTask.runInContext(),尤其是证书配置期间移除 shutdown-hook 的部分。在使用 ca.plugin.root.auth.strictness=true 的 KVM 主机上通过 provisionCertificate 重现问题,并跟踪 agent.log。当 shutdown 竞争不再导致 secured=false 被保留,同时成功的证书配置和重新连接流程仍得到保留时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
cloud, infrastructure, security
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
冷清
描述清晰度
描述清楚
新手友好度
72/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。