kubernetes-sigs / kubernetes-sigs/container-object-storage-interface

CI: track `object has been modified` instances in Controller and Sidecar

Open
#166 2 comments 0 reactions 0 assignees View on GitHub
api/v1alpha2 kind/cleanup priority/awaiting-more-evidence
Dominant language
Go
Stars
95
Forks
44
Avg merge
8h 12m
Merged PRs (30d)
4

Description

# Enhancement

**Why is this needed?**:

Instances of `object has been modified` errors when updating CRs/statuses indicate that mulitple actors are modifying the resource at once. Kubernetes and controller-runtime can easily handle these intances, and occasional occurrences are not reason for alarm. However, we should be on guard for cases where these errors pop up regularly. Frequently encountering these messages would mean that COSI controller/sidecar processes are interfering with each others' operations often, and COSI should limit this as much as reasonably possible.

**Describe the solution you'd like in detail**:

In order to help us gauge this, I would suggest that we create a CI test to look for this. These errors are more likely to occur when multiple CRs are created at once with references to each other. I would thus propose a test that creates a number of BucketClaims and referring BucketAccesses at once. In the test, we can wait for objects to reconcile, and then scan controller and sidecar logs for instances of `object has been modified` errors. If the number of instances passes a threshold (to be determined), fail the test.

Initially, we may want to set a somewhat high failure threshold. As we optimize the controllers to interfere less with each other, we can lower the failure threshold.

As a starting point, if the test creates 5 BucketClaims and 3 BucketAccesses, I would suggest 8 (5+3) total system occurrences of the error to be a failure. To me, this would indicate obvious interference that needs work.

As an end-goal for a well-optimized system, I think more than 2 instances would be a good failure threshold.

**Describe alternatives you've considered**:

We could aim for high optimization as a starting goal, but this would slow down development -- IMO, unnecessarily.

**Additional context**:

In my experience, errors like these are often raised by end users when they become too frequent or too predicable. Different users will have different ideas of what frequency/predictability is too much.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing CI tests and the Controller and Sidecar log collection used for reconciliation. Create concurrent BucketClaims and referring BucketAccesses, wait for reconciliation, and scan both logs for `object has been modified` occurrences. Done means the test applies a documented failure threshold and fails when the count exceeds it.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.