hashgraph / hashgraph/solo-weaver
feat(kubelet/crio): configure image GC thresholds and active disk-space reclamation
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 47
Description
## Problem
After provisioning and especially after block-node upgrades, leftover container images, stopped containers, and orphaned volumes accumulate on-disk and are **never reclaimed**. Operators have reported progressively shrinking free disk space with no obvious cause. Because neither kubelet nor CRI-O is given GC/eviction parameters, the default behaviour is to keep everything until manual intervention.
### Root-cause findings
A full audit of the codebase confirms the following gaps:
#### 1. Kubelet image GC thresholds — not set
`internal/templates/files/kubeadm/kubeadm-init.yaml` only injects `node-ip` into `kubeletExtraArgs`:
```yaml
kubeletExtraArgs:
- name: node-ip
value: {{.MachineIP}}
```
No `imageGCHighThresholdPercent`, `imageGCLowThresholdPercent`, `evictionHard`, `evictionSoft`, or `evictionMinimumReclaim` are set — anywhere in the template or in any `KubeletConfiguration` embed. Kubelet therefore uses its compiled-in defaults (85 % / 80 %), which may be too permissive for the relatively small disks common in node-operator deployments.
#### 2. CRI-O image GC thresholds — not set
`pkg/software/crio_installer.go :: generateExpectedCrioConfig()` (lines 657-692) writes only socket/path/log settings into the TOML config. There is no `imageGCHighThresholdPercent` or equivalent CRI-O knob configured, so CRI-O relies entirely on kubelet to trigger GC — which brings us back to gap 1.
#### 3. No active pruning step
No workflow step, daemon task, or script calls `crictl rmi`, `crictl rm`, `crio image rm`, or equivalent. The only reclamation that happens automatically is:
- `apt autoremove -y` at cluster-install time (`internal/workflows/setup.go:71`) — removes orphaned OS packages, not container images.
- `kubeadm reset --force` during full cluster teardown — not a routine maintenance action.
- `rm -rf /opt/solo/weaver` — a nuclear uninstall, not periodic cleanup.
#### 4. No disk-pressure monitoring
There is no proactive disk-usage check in any preflight, daemon loop, or upgrade workflow. Disk-space is only surfaced reactively in resolution hints (`"Check available disk space: df -h "`) on certain step failures.
#### 5. Potential orphaned resources after upgrades
Block-node upgrades replace Helm releases but do **not** guarantee that:
- Old image layers stored under `{sandboxDir}/var/lib/containers/storage/overlay` are evicted.
- PVCs created by prior Helm chart versions (tracked in `internal/blocknode/optional_storage.go`) are cleaned up if they fall outside the current chart's scope.
- Stopped/crashed containers from a failed upgrade are removed.
## Suggested fix scope
1. **Embed a `KubeletConfiguration` in `kubeadm-init.yaml`** (or via a `--config` drop-in) that explicitly sets:
- `imageGCHighThresholdPercent` — recommended 75–80 % for constrained disks
- `imageGCLowThresholdPercent` — recommended 60–70 %
- `evictionHard.nodefs.available` — e.g. `"10%"` / `"5Gi"`
- `evictionSoft.nodefs.available` + `evictionSoftGracePeriod` for a warning tier
2. **Set matching CRI-O GC thresholds** in `generateExpectedCrioConfig()` (`pkg/software/crio_installer.go:657`) so CRI-O participates in GC independently when kubelet is not signalling.
3. **Add a post-upgrade pruning step** in the block-node upgrade workflow that calls `crictl rmi --prune` (or the equivalent via the CRI-O API) to remove dangling/unused images after a successful upgrade.
4. **Add a preflight disk-space check** — before install and upgrade workflows begin, assert that free space on the sandbox partition meets a minimum threshold and emit a clear error with resolution hints if it doesn't.
5. **Consider a periodic daemon-side housekeeping task** (e.g. in `internal/daemon/`) that monitors disk usage under `{sandboxDir}/var/lib/containers/storage` and triggers `crictl rmi --prune` when usage exceeds a high-water mark.
## Affected files / packages
| File | Relevance |
|---|---|
| `internal/templates/files/kubeadm/kubeadm-init.yaml` | Needs `KubeletConfiguration` with GC/eviction settings |
| `pkg/software/crio_installer.go` `generateExpectedCrioConfig()` | Needs CRI-O GC threshold keys |
| `internal/workflows/steps/step_kubeadm.go` | Upgrade/init step — entry point for post-init GC |
| `internal/workflows/steps/step_block_node.go` | Post-upgrade pruning step should live here |
| `internal/daemon/` | Candidate location for periodic housekeeping |
| `internal/reality/` | Preflight disk-space check belongs here |
## Environment context
- CRI-O version: `v1.33.4` (`pkg/software/infrastructure-catalog.yaml:35`)
- Kubelet version: `v1.33.4` (`pkg/software/infrastructure-catalog.yaml:115`)
- Image storage root: `{sandboxDir}/var/lib/containers/storage` (non-standard path, separate from any system CRI-O install)
Contributor guide
Research direction
Start with internal/templates/files/kubeadm/kubeadm-init.yaml and generateExpectedCrioConfig() in pkg/software/crio_installer.go, then trace the upgrade and preflight entry points in internal/workflows/steps/step_kubeadm.go, internal/workflows/steps/step_block_node.go, and internal/reality/. Done means the requested kubelet and CRI-O thresholds, post-upgrade pruning, disk-space validation, and any selected daemon housekeeping behavior are implemented and exercised across the affected workflows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100