apache / apache/cloudstack

KVM agent PostCertificateRenewalTask fails with IllegalStateException during certificate provisioning

オープン
#12,795 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
component:agent component:kvm Severity:Minor type:bug
主要言語
Java
スター
3.1k
フォーク
1.4k
平均マージ
6日 19時間
マージ済み PR(30日)
32

説明

## Description

When `provisionCertificate` is called on a KVM host, the agent's `PostCertificateRenewalTask` consistently fails with `java.lang.IllegalStateException: Shutdown in progress`. This causes the host to permanently report `secured=false` in `StartupRoutingCommand.hostDetails`, showing as "Unsecure" in the UI despite having valid TLS certificates and a working SSL connection.

## Steps to Reproduce

1. Add a new KVM host to CloudStack 4.22 with `ca.plugin.root.auth.strictness=true`
2. Run `provisionCertificate hostid=`
3. The API returns `{"success": true}` — keystore, cert, CA cert, and key are all created correctly
4. The `PostCertificateRenewalTask` attempts to restart libvirtd and reconnect, but the cert provisioning triggers an agent restart
5. During the agent shutdown, `Runtime.removeShutdownHook()` throws `IllegalStateException: Shutdown in progress`
6. The agent never sets `secured=true` — the host permanently shows "Unsecure"

## Error Log (agent.log)

```
INFO [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Restarting libvirt after certificate provisioning/renewal
WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Execution of process for command [sudo service libvirtd restart ] failed.
WARN [cloud.agent.Agent] Failed to execute post certificate renewal command: java.lang.IllegalStateException: Shutdown in progress
at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
at com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1377)
```

## Root Cause

In `Agent.java:1377`, the `PostCertificateRenewalTask` calls `Runtime.getRuntime().removeShutdownHook()` during JVM shutdown, which is not allowed per Java spec. The certificate provisioning triggers an agent reconnect/restart, creating a race condition where the PostCertificateRenewal task runs during the shutdown window.

## Impact

- Host permanently shows "Unsecure" in the UI despite valid TLS
- The actual SSL connection works correctly (keystore loads, handshake succeeds)
- The `secured` flag in `host_details` DB table is overwritten to `false` on every agent reconnect
- Manual DB updates are overwritten by the agent's `StartupRoutingCommand`
- Reproduced consistently on multiple `provisionCertificate` attempts

## Suggested Fix

The `PostCertificateRenewalTask.runInContext()` should catch `IllegalStateException` from `removeShutdownHook()` and still proceed with setting the secured flag. Alternatively, check `Thread.currentThread().isInterrupted()` or use a guard flag before calling `removeShutdownHook()`.

```java
// In Agent.java PostCertificateRenewalTask.runInContext()
try {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
} catch (IllegalStateException e) {
// JVM is already shutting down, skip hook removal
LOG.debug("Skipping shutdown hook removal during shutdown", e);
}
```

## Environment

- CloudStack: 4.22.0.0
- OS: Ubuntu 22.04 (also reproduced on fresh provision)
- Java: OpenJDK 11.0.30 and 17.0.18
- KVM/libvirt: working correctly
- `ca.plugin.root.auth.strictness`: true

## Workaround

Manually update the DB: `UPDATE host_details SET value='true' WHERE host_id= AND name='secured';`
This is overwritten on next agent restart but the TLS connection is functionally secure regardless of the flag.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Agent.java の 1377 行付近から開始し、PostCertificateRenewalTask.runInContext() を調査します。特に、証明書のプロビジョニング中に shutdown-hook を削除する処理を確認してください。ca.plugin.root.auth.strictness=true を使用する KVM ホストで provisionCertificate により再現し、agent.log を追跡します。証明書のプロビジョニング成功と再接続フローを維持したまま、shutdown の競合によって secured=false が残らなくなれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
cloud, infrastructure, security
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
72/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。