Topology Provider permissions briefly dropped during reconciliation when the reflector watch resets

Affected Stackable version

Any up to and including SDP 26.3.0

Affected Apache HDFS version

N/A

Current and expected behavior

In the HDFS operator (and perhaps any operator based on kube-rs), when the Reflector watch resets, the Store has to be rebuilt.
Reconciliations before the Store is fully consistent can lead to service accounts being dropped from (Cluster)RoleBindings. The leads to the Topoology Provider not being able to determine the topology (or possibly builds an incorrect topology?)

The expected behaviour is that the above doesn't happen 😅.

Possible solution

We can requeue reconciliations (at least for some operations) until the store is fully consistent.

Eg:

• On error: log error and return early
• watcher::Event::Init -> the store is empty, waiting for InitApply events, requeue/return early.
• watcher::Event::InitApply -> store is partially populated, requeue/return early until InitDone.
• watcher::Event::InitDone -> store is populated, continue with reconcile
• watcher::Event::Apply -> store is populated, continue with reconcile
• watcher::Event::Delete -> store is populated, continue with reconcile

[!CAUTION]
I haven't checked to see whether we can and should requeue, or just return an error which bubbles up as a Result for the error_policy handler which logs and does requeues.
Regardless, we need to make sure it eventually is reconciled and not just ignored.

Additional context

My understanding of the problem/solution should be double checked with someone else.

[!TIP]
This might only be when the topology provider is used... but also seems like something that might affect other products that have components that interact with Kubernetes API in SDP generally

Environment

No response

Would you like to work on fixing this bug?

yes

Start by tracing the HDFS operator's reconciliation path and the kube-rs Reflector watcher event handling, especially Store rebuilds and RoleBinding updates. Check how Init, InitApply, InitDone, Apply, Delete, and error events currently affect reconciliation. Done means reconciliations cannot act on an empty or partially populated Store, service-account permissions are preserved, and the work is eventually requeued.