KusionStack / KusionStack/rollout
Rollout skips workload watchers when an unrelated API group returns a partial discovery error
- Dominant language
- Go
- Stars
- 21
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
## Description
The rollout controller fails to register workload watchers when `ServerGroupsAndResources()` returns a partial discovery error for an unrelated API group.
In our clusters, the failing API group is:
```text
metrics/v1alpha1: Got empty response for: metrics/v1alpha1
```
Although `apps.kusionstack.io/v1alpha1` is available, the controller skips registering watchers for `CollaSet`, `PodDecoration`, and `StatefulSet`.
## Environment
```text
Controller image: paascore-library/rollout:main-eb98574
Image digest: sha256:c11deb0b86ee667c00bd232d041042f0c2091895560ecfac3ede237bdf214549
Latest main commit: eb98574
```
The issue is still present in the latest `main` branch and in the source of the latest formal tag, `v0.2.0`.
## Controller Logs
```text
failed to get discovery result from member clusters, skip it
error="unable to retrieve the complete list of server APIs:
metrics/v1alpha1: Got empty response for: metrics/v1alpha1"
gvk="apps.kusionstack.io/v1alpha1, Kind=CollaSet"
```
The same error is logged for:
```text
apps.kusionstack.io/v1alpha1, Kind=PodDecoration
apps/v1, Kind=StatefulSet
```
## Steps to Reproduce
1. Configure a cluster where `CollaSet` is available.
2. Add an unrelated aggregated API that returns a discovery error, such as `metrics/v1alpha1`.
3. Start the rollout controller.
4. Create a Rollout that manages a CollaSet.
5. Update the CollaSet pod template.
## Actual Behavior
1. The admission webhook detects the pod template change.
2. It sets the CollaSet partition to the replica count.
3. The CollaSet update does not enqueue the associated Rollout.
4. No RolloutRun is created.
5. The workload remains stuck with:
```text
partition=replicas
currentRevision != updatedRevision
updatedReplicas=0
```
Updating an annotation on the Rollout immediately triggers reconciliation and allows the release to complete. This confirms that Rollout reconciliation works, but the CollaSet watcher was not registered.
## Expected Behavior
A discovery failure from an unrelated API group should not prevent supported workload watchers from being registered.
If `apps.kusionstack.io/v1alpha1` is successfully discovered, the controller should register the CollaSet watcher even when another API group fails discovery.
## Root Cause
The discovery implementation treats any error returned by `ServerGroupsAndResources()` as fatal:
```go
_, resources, err := d.client.ServerGroupsAndResources()
if err != nil {
return false, "", err
}
```
Kubernetes discovery can return both a partial resource list and a `GroupDiscoveryFailedError`. The current implementation discards the valid partial result.
`GetWatchableWorkloads()` then skips the workload:
```go
supported, msg, err := discoveryClient.IsSupported(gvk)
if err != nil {
logger.Error(
err,
"failed to get discovery result from member clusters, skip it",
"gvk",
gvk.String(),
)
return true
}
```
Because workload watchers are registered only during controller startup, they remain missing for the lifetime of the process.
## Git History
The partial-discovery error handling can be traced to:
```text
c4c97ac1c5592485a3f4894c99ffcdedbe1a5b5b
refactor(registry): unify interface and implementation code (#68)
```
The multi-cluster discovery implementation was later rewritten in:
```text
48115846d77c5e3e4922d8cb54ffc61267a667ff
feat: rewrite multicluster discovery and use Patch to update (#149)
```
The rewrite retained the same behavior.
## Proposed Fix
A fix is proposed in #164.
The pull request uses target-specific discovery instead of accepting or rejecting the aggregate result from all API groups. It checks the workload's target GroupVersion with `ServerResourcesForGroupVersion()`, so an unrelated discovery failure does not prevent a supported workload watcher from being registered.
The fix intentionally preserves the original safety boundaries:
- A missing target GVK is still treated as unsupported and its watcher is skipped.
- An error from the target GroupVersion still fails closed and its watcher is skipped.
- In multi-cluster mode, every current member cluster must support the target GVK.
- An unrelated API Group failure no longer vetoes an otherwise valid target GVK.
This is narrower than blindly ignoring every `GroupDiscoveryFailedError`: if the failed GroupVersion is the target itself, the controller must still skip registration.
## Suggested Tests
Add tests covering:
- An unrelated API group fails while the target CollaSet GVK is available.
- The target GVK itself fails discovery.
- One member cluster has an unrelated API discovery failure.
- Discovery fails completely without a usable partial result.
An unrelated API discovery failure should not prevent the CollaSet watcher from being registered.
## Operational Impact
The webhook successfully pauses the workload, but the controller never creates the corresponding RolloutRun. This leaves workloads permanently stuck at `partition=replicas`.
After fixing the discovery endpoint or controller implementation, the rollout-controller Pods must be restarted because workload watchers are registered only during startup.
Contributor guide
Research direction
Start by reading the discovery implementation around ServerGroupsAndResources, GetWatchableWorkloads, and IsSupported. Run or extend tests for unrelated, target, member-cluster, and completely failed discovery cases; done means valid target watchers register while target discovery failures still skip them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 30/100