canonical / canonical/charm-integration-testing
Validator fails with rc=127 when unit pod is rescheduled between setup and deep validation
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 96
Description
**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa
## Description
When a charm unit's pod is rescheduled between the validator setup phase (early `juju-exec` actions that install `/var/lib/validators/uv`) and the validation phase, the validator fails with `rc=127` because the setup files are no longer present on the fresh pod's ephemeral filesystem.
## Observed Behaviour
```
RuntimeError: Failed to create venv with python 3.10+ on neighbor/0 (rc=127):
/tmp/juju-exec3736521810/script.sh: line 1: /var/lib/validators/uv: No such file or directory
```
The validator installs `uv` at `/var/lib/validators/uv` via early `juju-exec` setup actions. When the pod is subsequently rescheduled by Kubernetes (ephemeral filesystem wiped), the validators assume `uv` is still present and fail with exit status 127.
## Timeline from Logs
| Time | Event |
|------|-------|
| `08:13:55Z` | `juju-exec` action — validator setup (installs `/var/lib/validators/uv`) |
| `08:14:46Z` | `juju-exec` action — second setup step |
| `08:16:03Z` | grafana-k8s pod receives SIGTERM → rescheduled by Kubernetes |
| `08:16:40Z` | Pod comes back; fresh filesystem — `/var/lib/validators/uv` is gone |
| `08:16:56Z` | Deep validator runs `juju-exec`, tries to use `/var/lib/validators/uv` → `rc=127` |
## Root Cause
The validator setup is not idempotent with respect to pod rescheduling. Files written to the pod's local filesystem (e.g., `/var/lib/validators/uv`) are lost when the pod is rescheduled. The deep validation phase does not check whether setup is still intact before proceeding, causing `exit status 127` (command not found).
## Suggested Fix
Before running deep validators, check whether setup is still intact on the target unit (e.g., verify `/var/lib/validators/uv` exists) and re-run the setup phase if any required files are missing. This would make validation resilient to pod rescheduling.
## Test Observer Context
- Artefact: `vault-k8s` rev 528, 1.19/candidate (artefact ID: 406378)
- Test plan: `integration/vault-k8s:vault-pki/tls-certificates/grafana-k8s:certificates`
- Test execution: https://test-observer.canonical.com/#/charms/406378?testExecutionId=458529&testResultId=10374026
- CI run: https://github.com/canonical/charm-integration-testing/actions/runs/24124435024
Contributor guide
Research direction
Trace the validator setup and deep-validation entry points described in issue 475, focusing on the juju-exec actions and the /var/lib/validators/uv dependency. Use the vault-k8s test plan and linked CI run to reproduce the rescheduling scenario. Done means validation handles a fresh pod without failing with rc=127.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100