kubernetes-sigs / kubernetes-sigs/multicluster-runtime

Coordinator ownership is not enforced for clusters reached outside a per-cluster Engage callback

Open
#172 2 comments 0 reactions 0 assignees View on GitHub
sig/multicluster
Dominant language
Go
Stars
311
Forks
54
Avg merge
5d 16h
Merged PRs (30d)
4

Description

### Problem

A `Coordinator` is meant to guarantee that only one replica in a fleet acts on a given cluster's resources. In practice, that guarantee only holds for one specific way of reaching a cluster. A replica that never won ownership of a cluster can still read and write that cluster's resources if it reaches the cluster by another route, such as a manual lookup or a watch that isn't itself cluster-specific. Multiple replicas can then reconcile the same cluster's resources at the same time, which defeats the purpose of running a Coordinator in the first place.

Two paths bypass the coordinator's guarantee that a replica only handles reconciliation requests for clusters it owns:

1. `Manager.GetCluster(ctx, name)` resolves directly through `provider.Get(ctx, name)`. It performs no ownership check. Any process that knows a cluster name receives a working client for it, whether or not the coordinator engaged that process for that cluster.
2. A raw source bound to a single, fixed cluster through `Source.ForCluster("", someCluster)` and `builder.WatchesRawSource(...)`. This pattern watches a cluster that isn't itself sharded (see "Context" below). Because the controller registers it once at startup, it runs on every replica unconditionally. `Coordinator.Engage` never gates it, and its handler can enqueue `mcreconcile.Requests` that name any cluster, including sharded clusters the process doesn't own.

### Context

For context, our operators run against two kinds of clusters at once:

- **Sharded clusters**: one control plane per tenant, discovered dynamically through a `multicluster.Provider`. Each corresponds to a tenant project in our system, but the library sees each only as a `multicluster.ClusterName`.
- **One shared cluster**: hosts our data-plane components. Controllers replicate selected resources from the sharded clusters into it and watch it back through a `Source.ForCluster` raw source, it's since one fixed cluster rather than something the provider discovers per tenant. This watch reflects status and drift on the shared cluster back into the sharded cluster that owns the corresponding resource.

The shared cluster's events map back to sharded ClusterName's the current replica may not own and both `GetCluster` and the reconcile dispatch return a working connection regardless.

### Reproduction

Environment: three replicas, `sharded.Coordinator` with `WithPerClusterLease(true)`, a `multicluster.Provider` backed by genuinely isolated per-cluster storage (ruling out a shared backend as a confound), and a controller with:

- a per-cluster watch (`mcbuilder.Watches`) on a sharded-cluster CR, correctly gated by the coordinator,
- a raw source on the one shared cluster (`Source.ForCluster("", sharedCluster)` plus `WatchesRawSource`), and
`mgr.GetCluster(ctx, req.ClusterName)` inside `Reconcile` to act on the sharded cluster identified in the request.

**Steps:**

1. Confirm leases are held for each cluster
1. Create or update a resource in one of the clusters. The owning replica reconciles it and writes a copy to the shared cluster. This step is correctly gated: only the owning replica will write the resource to the shared cluster.
1. That write fires the raw source's watch on all three replicas, since that watch is unconditional. Each replica's handler maps the event back to a request and calls `Reconcile`.
1. All three replicas' Reconcile calls succeed and process the object, including the two replicas never engaged for the cluster.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Manager.GetCluster(ctx, name), Coordinator.Engage, and the raw source registration through WatchesRawSource and reconcile dispatch. Reproduce the three-replica setup with per-cluster leases and a shared-cluster Source.ForCluster watch. Done means a replica that does not own a cluster cannot obtain or use its client or process requests for that cluster.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.