Keystore Password and CIFS Credentials Plaintext Exposure via SSH Command Logging in KVM CA Provisioning and Baremetal PXE

### Advisory Details

**Title**: Keystore Password and CIFS Credentials Plaintext Exposure via SSH Command Logging in KVM CA Provisioning and Baremetal PXE

**Description**:

### Summary

A critical sensitive information disclosure vulnerability (CWE-532 / CWE-209) exists in the Apache CloudStack Management Server due to flawed, position-dependent, and incomplete sanitization of SSH command logs. Highly sensitive credentials—specifically, dynamically-generated 16-character KVM keystore passwords and administrator-configured CIFS/Samba storage credentials—are logged in plaintext into the Management Server debug log files. Furthermore, during PXE command failures, the plaintext CIFS password is returned directly in the REST API error payload to the client.

---

### Details

The core command-logging desensitization mechanism in `SSHCmdHelper.java` employs a naive split-token sanitization method to prevent credentials from entering debug logs:
```java
LOGGER.debug("Executing cmd: " + cmd.split(KeyStoreUtils.KS_FILENAME)[0]);
```
Here, `KeyStoreUtils.KS_FILENAME` is the hardcoded string `"cloud.jks"`. This sanitization model assumes that any sensitive parameter in the command string will always follow the `"cloud.jks"` token. This design fails under two distinct reachable paths in the codebase:

#### 1. Core KVM Host CA Certificate Provisioning (`CAManagerImpl.java`)
During forced agent certificate updates via the `provisionCertificate` admin API, the certificate import shell command is generated as follows:
```java
final SetupCertificateCommand certificateCommand = new SetupCertificateCommand(certificate);
final SSHCmdHelper.SSHCmdResult setupCertResult = SSHCmdHelper.sshExecuteCmdWithResult(sshConnection,
String.format("sudo /usr/share/cloudstack-common/scripts/util/%s " +
"/etc/cloudstack/agent/agent.properties %s " +
"/etc/cloudstack/agent/%s %s " +
...
KeyStoreUtils.KS_IMPORT_SCRIPT,
keystorePassword, // <-- Positional parameter before KS_FILENAME
KeyStoreUtils.KS_FILENAME,
KeyStoreUtils.SSH_MODE,
...));
```
Because the dynamic plaintext `keystorePassword` is positioned *before* `"cloud.jks"`, `cmd.split("cloud.jks")[0]` evaluates to a string that still contains the plaintext password, causing it to be written directly into the debug logs.

#### 2. Baremetal PXE Template Operations (`BaremetalPingPxeResource.java`)
When setting up or backing up baremetal PXE resources, helper scripts like `prepare_tftp_bootfile.py` are executed via SSH with the CIFS storage server password (`_cifsPassword`) passed as an argument:
```java
String script =
String.format("python /usr/bin/prepare_tftp_bootfile.py restore %1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s %10$s %11$s", _tftpDir, cmd.getMac(),
_storageServer, _share, _dir, cmd.getTemplate(), _cifsUserName, _cifsPassword, cmd.getIp(), cmd.getNetMask(), cmd.getGateWay());
```
Because these command strings do not contain the `"cloud.jks"` token, the naive `split` operation evaluates to the entire original command, fully exposing the `_cifsPassword` in the logs. Furthermore, if the command fails, the exception returned to the caller (`PreparePxeServerAnswer` or `Answer`) appends the raw command containing the password, propagating it back in the REST API payload to the administrator.

---

### PoC

#### Prerequisites
- A deployed instance of the CloudStack Management Server.
- API Key and Secret Key for an administrator account.
- A KVM host or Baremetal PXE server configured in the workspace.

#### Reproduction Steps

1. Download the isolated container database and helper environment config from: [docker-compose.yml](https://gist.github.com/YLChen-007/4e7962a97463dc8ec75269b89b8c2107)
2. Download the active integration verification test script from: [verification_test.py](https://gist.github.com/YLChen-007/bb051953fbacf402d3b73a5f63029dee)
3. Download the baseline control test script from: [control-masked_output.py](https://gist.github.com/YLChen-007/f78a0a1ef2d4a0a839a3db885d612044)
4. Start the laboratory background services:
```bash
docker compose up -d
```
5. Execute the verification test to demonstrate the plaintext credential logging:
```bash
python3 verification_test.py
```
6. Execute the control test to verify that the split mechanism functions correctly under baseline conditions (i.e., when `cloud.jks` precedes the password):
```bash
python3 control-masked_output.py
```

---

### Log of Evidence

```text
=========================================================================
[*] Running Issue-cloudstack-12029 Keystore Password Exposure Verification
=========================================================================
[*] Stage 1: Attempting to connect to running CloudStack Management Server...
[!] CloudStack Management Server is offline (Expected in isolated environment).
[*] Switching to Academic/Logical Code-Path Verification...

--- Flow 1: CA Certificate Provisioning ---
Original constructed command:
sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties SecretRandomPassword123! /etc/cloudstack/agent/cloud.jks ssh /etc/cloudstack/agent/cloud.crt "CERT_DATA"
Logged command after split(KS_FILENAME)[0]:
sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties SecretRandomPassword123! /etc/cloudstack/agent/
🔴 [DEFECT-CONFIRMED] Plaintext password is fully exposed in Flow 1 log output!

--- Flow 2: Baremetal PXE template preparation ---
Original constructed command:
python /usr/bin/prepare_tftp_bootfile.py restore /var/lib/tftpboot 00:11:22:33:44:55 192.168.1.100 share_name /tftp_dir template_name admin_user CifsSecretPassword999! 192.168.1.101 255.255.255.0 192.168.1.254
Logged command after split(KS_FILENAME)[0]:
python /usr/bin/prepare_tftp_bootfile.py restore /var/lib/tftpboot 00:11:22:33:44:55 192.168.1.100 share_name /tftp_dir template_name admin_user CifsSecretPassword999! 192.168.1.101 255.255.255.0 192.168.1.254
🔴 [DEFECT-CONFIRMED] Plaintext CIFS password is fully exposed in Flow 2 log output!

=========================================================================
RESULT: DEFECT-CONFIRMED (TRUE POSITIVE)
=========================================================================
=========================================================================
[*] Running Issue-cloudstack-12029 Control Group Test: Masked Output Baseline
=========================================================================
Control group command (cloud.jks placed before password):
sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/cloud.jks SecretRandomPassword123! 365 /etc/cloudstack/agent/cloud.csr
Logged command after split(KS_FILENAME)[0]:
sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/

--- Control Group Verification ---
🟢 [CONTROL-SUCCESS] Plaintext password is successfully masked/omitted from the log!
The security mechanism functions correctly under normal/baseline conditions.
=========================================================================
```

---

### Impact

- **Vulnerability Type**: CWE-532 (Insertion of Sensitive Information into Log File) / CWE-209 (Generation of Error Message Containing Sensitive Information)
- **Compromised Assets**: KVM host CA keystore private keys, administrative CIFS/Samba storage credentials.
- **Severity Impact**: High. Exposure of keystore passwords allows remote actors with system read access to decrypt agent TLS private keys and compromise hypervisor agent communication (e.g. performing MITM and executing arbitrary VM manipulation commands). Exposure of CIFS credentials gives complete read-write access to private cloud system template repositories on the corporate network.

---

### Affected products

- **Ecosystem**: maven
- **Package name**: org.apache.cloudstack:cloudstack
- **Affected versions**: `<= 4.22.1.0` (up to unreleased pre-release state on the `main` branch)
- **Patched versions**:

---

### Severity

- **Severity**: High
- **Vector string**: CVSS:3.1/AV:N/AC:L/PR:H/UI:N/S:U/C:H/I:H/A:N

---

### Weaknesses

- **CWE**: CWE-532: Insertion of Sensitive Information into Log File
- **CWE**: CWE-209: Generation of Error Message Containing Sensitive Information

---

### Occurrences

| Permalink | Description |
| :--- | :--- |
| [utils/src/main/java/com/cloud/utils/ssh/SSHCmdHelper.java#L167](https://github.com/apache/cloudstack/blob/3f6866d70a6e4eb8a12dd257d33bde5aae79aacf/utils/src/main/java/com/cloud/utils/ssh/SSHCmdHelper.java#L167) | Naive command sanitization using `cmd.split(KeyStoreUtils.KS_FILENAME)[0]` in `sshExecuteCmdOneShot()`. |
| [utils/src/main/java/com/cloud/utils/ssh/SSHCmdHelper.java#L230](https://github.com/apache/cloudstack/blob/3f6866d70a6e4eb8a12dd257d33bde5aae79aacf/utils/src/main/java/com/cloud/utils/ssh/SSHCmdHelper.java#L230) | Naive command sanitization in the standard output/error logging statement in `sshExecuteCmdOneShot()`. |
| [server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java#L278-L294](https://github.com/apache/cloudstack/blob/3f6866d70a6e4eb8a12dd257d33bde5aae79aacf/server/src/main/java/org/apache/cloudstack/ca/CAManagerImpl.java#L278-L294) | Certificate import command creation in `provisionCertificateViaSsh()`, positioning `keystorePassword` before the split-token. |
| [plugins/hypervisors/baremetal/src/main/java/com/cloud/baremetal/networkservice/BaremetalPingPxeResource.java#L155-L160](https://github.com/apache/cloudstack/blob/3f6866d70a6e4eb8a12dd257d33bde5aae79aacf/plugins/hypervisors/baremetal/src/main/java/com/cloud/baremetal/networkservice/BaremetalPingPxeResource.java#L155-L160) | Construction and execution of Baremetal PXE TFTP restore command passing `_cifsPassword` without split-token and returning it in answer payload on failure. |
| [plugins/hypervisors/baremetal/src/main/java/com/cloud/baremetal/networkservice/BaremetalPingPxeResource.java#L184-L189](https://github.com/apache/cloudstack/blob/3f6866d70a6e4eb8a12dd257d33bde5aae79aacf/plugins/hypervisors/baremetal/src/main/java/com/cloud/baremetal/networkservice/BaremetalPingPxeResource.java#L184-L189) | Construction and execution of Baremetal PXE TFTP backup command passing `_cifsPassword` without split-token and returning it in answer payload on failure. |

Beginne mit SSHCmdHelper.java und verfolge sshExecuteCmdOneShot() nach. Untersuche anschließend die Konstruktion von Befehlen und die Fehlerbehandlung in CAManagerImpl.java und BaremetalPingPxeResource.java. Verwende verification_test.py und control-masked_output.py, um das Logging-Verhalten zu reproduzieren. Erledigt bedeutet, dass keystore- und CIFS-Anmeldedaten in Debug-Logs sowie in PXE-Fehlerantworten nicht vorhanden sind.

Vom Indexierungsmodell aus dem Issue-Text verfasst.