bug: .spec.clusterOperation.stopped doesn't consider dependence, leads to Brokers hanging on shutdown while trying to reach controllers

Fix: Kafka hangs on shutdown when dependencies are terminated simultaneously

Problem

When a KafkaCluster is stopped (via clusterOperation.stopped: true or namespace deletion),
all roles (brokers, controllers/ZooKeeper) are terminated simultaneously. Brokers have
controlled.shutdown.enable=true, which requires communication with controllers (KRaft) or
ZooKeeper during shutdown. If those are already gone, brokers block until the
terminationGracePeriodSeconds expires (default 30 minutes).

Current behavior

• ClusterOperation is cluster-wide only (crd/mod.rs:138) - no per-role control.
• ClusterResourceApplyStrategy::from(&kafka.spec.cluster_operation) (kafka_controller.rs:282)
  applies uniformly to all StatefulSets in one reconciliation pass.
• Both broker and controller StatefulSets use podManagementPolicy: Parallel
  (statefulset.rs:550, :877), so all pods terminate at once.
• The reconciler iterates roles in a single loop (kafka_controller.rs:352-488) with no
  ordering or phasing.

Option A: Framework-level support in operator-rs

Add a concept of role shutdown ordering to the ClusterResourceApplyStrategy / ClusterResources
framework in operator-rs, so any operator (Kafka, Druid, Trino, etc.) can declare dependency
ordering between roles.

Approach

1. Extend ClusterOperation or ClusterResources with role dependency metadata.
   The operator would declare something like:

   cluster_resources.set_shutdown_order(vec![
       KafkaRole::Broker,    // stop first
       KafkaRole::Controller, // stop after brokers are gone
   ]);
   

2. Multi-phase reconciliation in ClusterResources::add().
   When the strategy is "stopped", cluster_resources.add() for a StatefulSet would:

   • Phase 1: Set replicas to 0 only for the first role in the shutdown order.
   • Phase 2: On the next reconciliation (triggered by the StatefulSet watch), check that all
     phase-1 StatefulSets have 0 ready replicas, then set replicas to 0 for the next role.
   • Repeat until all roles are stopped.

3. Condition reporting per phase.
   Add conditions like StoppingSince or ShutdownPhase: brokers so the status reflects
   progress.

Pros

• Reusable across all Stackable operators (any product with role dependencies).
• Clean separation of concerns - individual operators just declare the ordering.
• Handles both stopped: true and namespace deletion (if the operator gets a chance to reconcile).

Cons

• Larger scope - requires changes to the shared operator-rs crate.
• Multi-phase reconciliation adds complexity to the framework.
• Does NOT help with namespace deletion if all resources are deleted simultaneously
  (Kubernetes may not give the operator time to reconcile between phases).

Option B: Custom sequencing in the Kafka operator reconciler

Handle shutdown ordering directly in kafka_controller.rs by splitting the reconciliation
into phases when stopped: true.

Approach

1. Detect the stopped state early in the reconciler (kafka_controller.rs, before the
   role loop at line 352).

2. Check current StatefulSet state before applying.
   When cluster_operation.stopped == true:

   // Pseudo-code
   let broker_sts_list = list_statefulsets(label: component=broker);
   let all_brokers_stopped = broker_sts_list.iter().all(|sts| sts.status.replicas == 0);
   
   if !all_brokers_stopped {
       // Only scale down brokers, leave controllers untouched
       // Apply broker StatefulSets with replicas: 0
       // Apply controller StatefulSets with their current replica count
       // Requeue after a short delay
       return Ok(Action::requeue(Duration::from_secs(5)));
   }
   
   // All brokers are gone, now scale down controllers
   // Apply all StatefulSets with replicas: 0
   

3. Modify the role loop to accept a per-role override for the apply strategy, or
   skip applying certain StatefulSets depending on the shutdown phase.

4. Apply the same logic in reverse for startup (optional): when transitioning from
   stopped: true to stopped: false, start controllers first, wait for them to be ready,
   then start brokers.

Pros

• Self-contained in the Kafka operator - no upstream changes needed.
• Can ship independently on the Kafka operator's release cycle.
• Simpler to reason about since the logic is Kafka-specific.

Cons

• Not reusable - other operators with similar problems (e.g., Druid, HBase) would need their
  own implementations.
• Adds complexity to the Kafka reconciler, which is already non-trivial.
• Still does NOT help with namespace deletion (same limitation as Option A).