stackabletech / stackabletech/zookeeper-operator

ZooKeeper znode controller: release finalizer without connecting when the parent is deleting

Open
#1,049 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
37
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
10

Description

Problem

In some scenarios, the ZookeeperZnode finalizer (zookeeper.stackable.tech/znode) can take around 130–150s to release. That's what sometimes leaves namespaces stuck in Terminating in our integration tests.

This can happen when the operator tries connection to Zookeeper to delete the node, while ZooKeeper is being torn down, the delete/cleanup path (ensure_znode_missing) runs its errors through controller-runtime's exponential backoff, and that's where the multi-minute stall comes from.

Mechanism

  1. znode cleanup (ensure_znode_missing) tries to connect to the ZK service and gets Connection refused — the ZK pods/endpoints are already gone — so the reconcile errors out.
  2. The ZookeeperCluster CR then drops out of the watch cache. At this point the finalizer's fast path (cluster doesn't exist → assume the znode is gone → drop the finalizer without connecting) would kick in, but the failed reconcile is already sitting in exponential backoff.
  3. ~139s gap: nothing re-runs, even though the fast-path condition is now true.
  4. Backoff finally expires, the reconcile re-runs, the fast path fires, the finalizer is removed, and the namespace deletes.

So it comes down to queue ordering under load. If cleanup runs after the CR leaves the store, it's instant if it runs before, it errors, hits backoff, and takes 130s+.

Fix

In the finalizer::Event::Cleanup arm: if the referenced ZookeeperCluster has a deletionTimestamp, drop the finalizer straight away without connecting to ZK. Retrying an unreachable server makes sense on the create path; on the delete path it shouldn't be allowed to block teardown.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the finalizer::Event::Cleanup arm and trace how it handles a referenced ZookeeperCluster with a deletionTimestamp. Check the ensure_znode_missing cleanup path and the integration tests that expose namespaces stuck in Terminating. Done means deletion skips the ZooKeeper connection and releases the finalizer without the multi-minute backoff.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.