canonical / canonical/charm-integration-testing

Validator fails with rc=127 when unit pod is rescheduled between setup and deep validation

Open
#475 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6
Forks
1
Avg merge
1d 22h
Merged PRs (30d)
96

Description

**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa

## Description

When a charm unit's pod is rescheduled between the validator setup phase (early `juju-exec` actions that install `/var/lib/validators/uv`) and the validation phase, the validator fails with `rc=127` because the setup files are no longer present on the fresh pod's ephemeral filesystem.

## Observed Behaviour

```
RuntimeError: Failed to create venv with python 3.10+ on neighbor/0 (rc=127):
/tmp/juju-exec3736521810/script.sh: line 1: /var/lib/validators/uv: No such file or directory
```

The validator installs `uv` at `/var/lib/validators/uv` via early `juju-exec` setup actions. When the pod is subsequently rescheduled by Kubernetes (ephemeral filesystem wiped), the validators assume `uv` is still present and fail with exit status 127.

## Timeline from Logs

| Time | Event |
|------|-------|
| `08:13:55Z` | `juju-exec` action — validator setup (installs `/var/lib/validators/uv`) |
| `08:14:46Z` | `juju-exec` action — second setup step |
| `08:16:03Z` | grafana-k8s pod receives SIGTERM → rescheduled by Kubernetes |
| `08:16:40Z` | Pod comes back; fresh filesystem — `/var/lib/validators/uv` is gone |
| `08:16:56Z` | Deep validator runs `juju-exec`, tries to use `/var/lib/validators/uv` → `rc=127` |

## Root Cause

The validator setup is not idempotent with respect to pod rescheduling. Files written to the pod's local filesystem (e.g., `/var/lib/validators/uv`) are lost when the pod is rescheduled. The deep validation phase does not check whether setup is still intact before proceeding, causing `exit status 127` (command not found).

## Suggested Fix

Before running deep validators, check whether setup is still intact on the target unit (e.g., verify `/var/lib/validators/uv` exists) and re-run the setup phase if any required files are missing. This would make validation resilient to pod rescheduling.

## Test Observer Context

- Artefact: `vault-k8s` rev 528, 1.19/candidate (artefact ID: 406378)
- Test plan: `integration/vault-k8s:vault-pki/tls-certificates/grafana-k8s:certificates`
- Test execution: https://test-observer.canonical.com/#/charms/406378?testExecutionId=458529&testResultId=10374026
- CI run: https://github.com/canonical/charm-integration-testing/actions/runs/24124435024

Contributor guide

Open the contributing guide

Research direction

Trace the validator setup and deep-validation entry points described in issue 475, focusing on the juju-exec actions and the /var/lib/validators/uv dependency. Use the vault-k8s test plan and linked CI run to reproduce the rescheduling scenario. Done means validation handles a fresh pod without failing with rc=127.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.