hashgraph / hashgraph/solo-weaver

feat(kubelet/crio): configure image GC thresholds and active disk-space reclamation

Open
#890 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3
Forks
0
Avg merge
3d 5h
Merged PRs (30d)
47

Description

## Problem

After provisioning and especially after block-node upgrades, leftover container images, stopped containers, and orphaned volumes accumulate on-disk and are **never reclaimed**. Operators have reported progressively shrinking free disk space with no obvious cause. Because neither kubelet nor CRI-O is given GC/eviction parameters, the default behaviour is to keep everything until manual intervention.

### Root-cause findings

A full audit of the codebase confirms the following gaps:

#### 1. Kubelet image GC thresholds — not set

`internal/templates/files/kubeadm/kubeadm-init.yaml` only injects `node-ip` into `kubeletExtraArgs`:

```yaml
kubeletExtraArgs:
- name: node-ip
value: {{.MachineIP}}
```

No `imageGCHighThresholdPercent`, `imageGCLowThresholdPercent`, `evictionHard`, `evictionSoft`, or `evictionMinimumReclaim` are set — anywhere in the template or in any `KubeletConfiguration` embed. Kubelet therefore uses its compiled-in defaults (85 % / 80 %), which may be too permissive for the relatively small disks common in node-operator deployments.

#### 2. CRI-O image GC thresholds — not set

`pkg/software/crio_installer.go :: generateExpectedCrioConfig()` (lines 657-692) writes only socket/path/log settings into the TOML config. There is no `imageGCHighThresholdPercent` or equivalent CRI-O knob configured, so CRI-O relies entirely on kubelet to trigger GC — which brings us back to gap 1.

#### 3. No active pruning step

No workflow step, daemon task, or script calls `crictl rmi`, `crictl rm`, `crio image rm`, or equivalent. The only reclamation that happens automatically is:

- `apt autoremove -y` at cluster-install time (`internal/workflows/setup.go:71`) — removes orphaned OS packages, not container images.
- `kubeadm reset --force` during full cluster teardown — not a routine maintenance action.
- `rm -rf /opt/solo/weaver` — a nuclear uninstall, not periodic cleanup.

#### 4. No disk-pressure monitoring

There is no proactive disk-usage check in any preflight, daemon loop, or upgrade workflow. Disk-space is only surfaced reactively in resolution hints (`"Check available disk space: df -h "`) on certain step failures.

#### 5. Potential orphaned resources after upgrades

Block-node upgrades replace Helm releases but do **not** guarantee that:
- Old image layers stored under `{sandboxDir}/var/lib/containers/storage/overlay` are evicted.
- PVCs created by prior Helm chart versions (tracked in `internal/blocknode/optional_storage.go`) are cleaned up if they fall outside the current chart's scope.
- Stopped/crashed containers from a failed upgrade are removed.

## Suggested fix scope

1. **Embed a `KubeletConfiguration` in `kubeadm-init.yaml`** (or via a `--config` drop-in) that explicitly sets:
- `imageGCHighThresholdPercent` — recommended 75–80 % for constrained disks
- `imageGCLowThresholdPercent` — recommended 60–70 %
- `evictionHard.nodefs.available` — e.g. `"10%"` / `"5Gi"`
- `evictionSoft.nodefs.available` + `evictionSoftGracePeriod` for a warning tier

2. **Set matching CRI-O GC thresholds** in `generateExpectedCrioConfig()` (`pkg/software/crio_installer.go:657`) so CRI-O participates in GC independently when kubelet is not signalling.

3. **Add a post-upgrade pruning step** in the block-node upgrade workflow that calls `crictl rmi --prune` (or the equivalent via the CRI-O API) to remove dangling/unused images after a successful upgrade.

4. **Add a preflight disk-space check** — before install and upgrade workflows begin, assert that free space on the sandbox partition meets a minimum threshold and emit a clear error with resolution hints if it doesn't.

5. **Consider a periodic daemon-side housekeeping task** (e.g. in `internal/daemon/`) that monitors disk usage under `{sandboxDir}/var/lib/containers/storage` and triggers `crictl rmi --prune` when usage exceeds a high-water mark.

## Affected files / packages

| File | Relevance |
|---|---|
| `internal/templates/files/kubeadm/kubeadm-init.yaml` | Needs `KubeletConfiguration` with GC/eviction settings |
| `pkg/software/crio_installer.go` `generateExpectedCrioConfig()` | Needs CRI-O GC threshold keys |
| `internal/workflows/steps/step_kubeadm.go` | Upgrade/init step — entry point for post-init GC |
| `internal/workflows/steps/step_block_node.go` | Post-upgrade pruning step should live here |
| `internal/daemon/` | Candidate location for periodic housekeeping |
| `internal/reality/` | Preflight disk-space check belongs here |

## Environment context

- CRI-O version: `v1.33.4` (`pkg/software/infrastructure-catalog.yaml:35`)
- Kubelet version: `v1.33.4` (`pkg/software/infrastructure-catalog.yaml:115`)
- Image storage root: `{sandboxDir}/var/lib/containers/storage` (non-standard path, separate from any system CRI-O install)

Contributor guide

Open the contributing guide

Research direction

Start with internal/templates/files/kubeadm/kubeadm-init.yaml and generateExpectedCrioConfig() in pkg/software/crio_installer.go, then trace the upgrade and preflight entry points in internal/workflows/steps/step_kubeadm.go, internal/workflows/steps/step_block_node.go, and internal/reality/. Done means the requested kubelet and CRI-O thresholds, post-upgrade pruning, disk-space validation, and any selected daemon housekeeping behavior are implemented and exercised across the affected workflows.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.