canonical / canonical/data-platform-libs

`storage-detaching` hook fails on Juju 3.5 due to secrets creation for dying units

Open
#243 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
13
Forks
12
PR merge metrics
No merged PRs in 30d

Description

## Steps to reproduce

1. Clone the source code for `cassandra-operator`.
If this issue is reviewed after a temporary bug fix on the operator side, remove the following lines from `src/core/state.py`:

```python
if (self.model.juju_version <= JujuVersion("3.5.7")) and os.getenv("JUJU_HOOK_NAME", "") == f"{DATA_STORAGE}-storage-detaching":
unit_additional_secret_fields = None
```

2. Build and deploy 2 units:

```bash
juju deploy ./cassandra_ubuntu@24.04-amd64.charm cassandra --config profile=testing --num-units=2
```

3. Remove a secondary unit:

```bash
juju remove-unit
```

4. Wait until the `storage-detaching` hook completes on the departing unit.

5. Observe that the hook fails on **any remaining unit** with the error:

```
cannot apply changes: creating secrets: cannot create secret for owner "" which is not alive
```

## Expected behavior

* After a unit begins removal (life = `dying`), **no new secrets should be created** on its behalf, neither by the departing unit nor by surviving peers.
* `DataPeerUnitData` should **skip secret creation** for units whose life is `dying` or **defer until the unit is alive again**.
* The unit should be removed cleanly after the `storage-detaching` hook completes successfully.

## Actual behavior

* `DataPeerUnitData` automatically attempts to create Juju secrets for all fields in `additional_secret_fields`.
* Juju rejects requests for units that are not alive, causing the hook to **fail and retry indefinitely**, which blocks the application.

## Versions

Operating system: Ubuntu 24.04.1 LTS

Juju CLI: 3.5.7-genericlinux-amd64

Juju agent: 3.5.7

## Log output

```
unit-cassandra-1: 17:10:48 ERROR juju.worker.uniter.context cannot apply changes: creating secrets: cannot create secret for owner "unit-cassandra-1" which is not alive
unit-cassandra-1: 17:10:48 ERROR juju.worker.uniter.operation hook "data-storage-detaching" (via hook dispatching script: dispatch) failed: creating secrets: cannot create secret for owner "unit-cassandra-1" which is not alive
```

## Additional context

* This issue occurs **only on Juju v3.5**; Juju v3.6 does not have this problem.
* The issue can be worked around by **removing `additional_secret_fields`** from `DataPeerUnitData` during unit removal:

```python
if (self.model.juju_version <= JujuVersion("3.5.7")) and os.getenv("JUJU_HOOK_NAME", "") == f"{DATA_STORAGE}-storage-detaching":
unit_additional_secret_fields = None
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.