kubernetes-sigs / kubernetes-sigs/cli-utils

Bug: Inventory updates should tolerate drift (and overwrite it)

Open
#559 4 comments 0 reactions 0 assignees View on GitHub
lifecycle/frozen
Dominant language
Go
Stars
178
Forks
85
PR merge metrics
No merged PRs in 30d

Description

Right now, inventory updates may return a conflict error from Kubernetes. The inventory client should detect this (`apierrors.IsConflict(err)`) and retry with a new Get (to update the ResourceVersion) + Update.

Example retry code:

```
type retriable func(ctx context.Context) (retry bool, err error)

func retryWithBackoff(ctx context.Context, timeout time.Duration, fn retriable) error {
var err error
var retry bool
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
delay := 1 + time.Second
for {
// attempt to update status
retry, err = fn(ctx)
if !retry {
return err
}

// wait until delay or timeout
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return fmt.Errorf("timed out after retrying for %v: %w", timeout, err)
case <-timer.C:
// continue
}
// retry backoff
delay = delay * 2
}
}
```

example usage:
```
// attempt to update status until timeout
ctx := context.TODO()
timeout := 1 * time.Minute
return retryWithBackoff(ctx, timeout, func(ctx context.Context) (retry bool, err error) {
// Get the object to get the latest ResourceVersion.
latestObj, err := resource.Get(ctx, obj.GetName(), metav1.GetOptions{TypeMeta: meta})
if err != nil {
return false, fmt.Errorf("failed to get inventory status from cluster: %w", err)
}
// Ignore any status changes made remotely.
// This update will replace them.
obj.SetResourceVersion(latestObj.GetResourceVersion())

_, err = resource.UpdateStatus(ctx, obj, metav1.UpdateOptions{TypeMeta: meta})
if err != nil {
// retry if conflict
return apierrors.IsConflict(err), fmt.Errorf("failed to write updated inventory status to cluster: %w", err)
}
return false, nil
})
```

Another option is to use https://github.com/flowchartsman/retry which is nice and generic. gcloud and client-go also have retry libs.

Contributor guide

Open the contributing guide

Research direction

No file or test is named; begin by locating the inventory client's UpdateStatus path and the existing Kubernetes resource update handling. Verify that a conflict causes a fresh Get and retry until success or timeout, while non-conflict errors remain errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.