apache / apache/cloudstack

KVM agent PostCertificateRenewalTask fails with IllegalStateException during certificate provisioning

Aberta
#12,795 1 comentário 0 reações 0 responsáveis Ver no GitHub
component:agent component:kvm Severity:Minor type:bug
Linguagem predominante
Java
Estrelas
3.1k
Forks
1.4k
Merge médio
6d 19h
PRs com merge (30d)
32

Descrição

## Description

When `provisionCertificate` is called on a KVM host, the agent's `PostCertificateRenewalTask` consistently fails with `java.lang.IllegalStateException: Shutdown in progress`. This causes the host to permanently report `secured=false` in `StartupRoutingCommand.hostDetails`, showing as "Unsecure" in the UI despite having valid TLS certificates and a working SSL connection.

## Steps to Reproduce

1. Add a new KVM host to CloudStack 4.22 with `ca.plugin.root.auth.strictness=true`
2. Run `provisionCertificate hostid=`
3. The API returns `{"success": true}` — keystore, cert, CA cert, and key are all created correctly
4. The `PostCertificateRenewalTask` attempts to restart libvirtd and reconnect, but the cert provisioning triggers an agent restart
5. During the agent shutdown, `Runtime.removeShutdownHook()` throws `IllegalStateException: Shutdown in progress`
6. The agent never sets `secured=true` — the host permanently shows "Unsecure"

## Error Log (agent.log)

```
INFO [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Restarting libvirt after certificate provisioning/renewal
WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Execution of process for command [sudo service libvirtd restart ] failed.
WARN [cloud.agent.Agent] Failed to execute post certificate renewal command: java.lang.IllegalStateException: Shutdown in progress
at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
at com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1377)
```

## Root Cause

In `Agent.java:1377`, the `PostCertificateRenewalTask` calls `Runtime.getRuntime().removeShutdownHook()` during JVM shutdown, which is not allowed per Java spec. The certificate provisioning triggers an agent reconnect/restart, creating a race condition where the PostCertificateRenewal task runs during the shutdown window.

## Impact

- Host permanently shows "Unsecure" in the UI despite valid TLS
- The actual SSL connection works correctly (keystore loads, handshake succeeds)
- The `secured` flag in `host_details` DB table is overwritten to `false` on every agent reconnect
- Manual DB updates are overwritten by the agent's `StartupRoutingCommand`
- Reproduced consistently on multiple `provisionCertificate` attempts

## Suggested Fix

The `PostCertificateRenewalTask.runInContext()` should catch `IllegalStateException` from `removeShutdownHook()` and still proceed with setting the secured flag. Alternatively, check `Thread.currentThread().isInterrupted()` or use a guard flag before calling `removeShutdownHook()`.

```java
// In Agent.java PostCertificateRenewalTask.runInContext()
try {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
} catch (IllegalStateException e) {
// JVM is already shutting down, skip hook removal
LOG.debug("Skipping shutdown hook removal during shutdown", e);
}
```

## Environment

- CloudStack: 4.22.0.0
- OS: Ubuntu 22.04 (also reproduced on fresh provision)
- Java: OpenJDK 11.0.30 and 17.0.18
- KVM/libvirt: working correctly
- `ca.plugin.root.auth.strictness`: true

## Workaround

Manually update the DB: `UPDATE host_details SET value='true' WHERE host_id= AND name='secured';`
This is overwritten on next agent restart but the TLS connection is functionally secure regardless of the flag.

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Comece em Agent.java por volta da linha 1377 e inspecione PostCertificateRenewalTask.runInContext(), especialmente a remoção do shutdown-hook durante o provisionamento do certificado. Reproduza o problema com provisionCertificate em um host KVM usando ca.plugin.root.auth.strictness=true e acompanhe agent.log. Está concluído quando a condição de corrida de shutdown não deixar mais secured=false, preservando o provisionamento bem-sucedido do certificado e o fluxo de reconexão.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
java
Domínio
cloud, infrastructure, security
Tipo de issue
Bug
Dificuldade
3/5
Tempo estimado
1-2 dias
Status de atividade
Pouca atividade
Clareza
Claramente especificada
Facilidade para iniciantes
72/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.