kubeslice / kubeslice/worker-operator

Bug: `retryAttempts` is a package-level variable — shared across all Cluster objects and survives controller restarts

Open
#484 0 comments 0 reactions 4 assignees Claimed by @richiesebastian View on GitHub
bug
Dominant language
Go
Stars
62
Forks
33
Avg merge
3d 10h
Merged PRs (30d)
3

Description

### 📜 Description

`pkg/hub/controllers/cluster/reconciler.go` declares `var retryAttempts = 0` at the package level (line 88). The `handleClusterDeletion` function increments it to enforce `MAX_CLUSTER_DEREGISTRATION_ATTEMPTS = 3`. Because it is a package-level var:

- If multiple Cluster objects are being deleted simultaneously, their attempt counts are merged.
- After 3 failures on any single cluster, all subsequent clusters also stop retrying immediately.
- Restarting the operator pod resets the counter, breaking the intended "stop after 3 attempts" semantics.

### 👟 Reproduction steps

1. Create two Cluster CRs.
2. Make `createDeregisterJob` fail (mock or break the job).
3. Delete both clusters simultaneously.
4. Observe that the first cluster's failures exhaust retryAttempts for the second.

### 👍 Expected behavior

Each Cluster object should track its own deregistration attempt count — store it in the Cluster CR's status or annotations.

### 👎 Actual Behavior

Shared global counter causes incorrect retry limiting across unrelated cluster objects.

### 🐚 Relevant log output

```shell

```

### Version

_No response_

### 🖥️ What operating system are you seeing the problem on?

_No response_

### ✅ Proposed Solution

Move attempt tracking into the Cluster CR's status field or use an annotation. Alternatively, use `r.Status().Update` to persist a retry count per object.

### 👀 Have you spent some time to check if this issue has been raised before?

- [x] I checked and didn't find any similar issue

### Code of Conduct

- [x] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.