ZooKeeper znode controller: release finalizer without connecting when the parent is deleting

Abierto
#1,049 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
68/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
kubernetes, rust
Área
devops, distributed-systems

Línea de trabajo

Comienza en la rama finalizer::Event::Cleanup y sigue cómo gestiona un ZookeeperCluster referenciado con un deletionTimestamp. Revisa la ruta de limpieza ensure_znode_missing y las pruebas de integración que muestran namespaces atascados en Terminating. Se considera completado cuando la eliminación omite la conexión de ZooKeeper y libera el finalizer sin el backoff de varios minutos.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

In some scenarios, the ZookeeperZnode finalizer (zookeeper.stackable.tech/znode) can take around 130–150s to release. That's what sometimes leaves namespaces stuck in Terminating in our integration tests.

This can happen when the operator tries connection to Zookeeper to delete the node, while ZooKeeper is being torn down, the delete/cleanup path (ensure_znode_missing) runs its errors through controller-runtime's exponential backoff, and that's where the multi-minute stall comes from.

Mechanism

  1. znode cleanup (ensure_znode_missing) tries to connect to the ZK service and gets Connection refused — the ZK pods/endpoints are already gone — so the reconcile errors out.
  2. The ZookeeperCluster CR then drops out of the watch cache. At this point the finalizer's fast path (cluster doesn't exist → assume the znode is gone → drop the finalizer without connecting) would kick in, but the failed reconcile is already sitting in exponential backoff.
  3. ~139s gap: nothing re-runs, even though the fast-path condition is now true.
  4. Backoff finally expires, the reconcile re-runs, the fast path fires, the finalizer is removed, and the namespace deletes.

So it comes down to queue ordering under load. If cleanup runs after the CR leaves the store, it's instant if it runs before, it errors, hits backoff, and takes 130s+.

Fix

In the finalizer::Event::Cleanup arm: if the referenced ZookeeperCluster has a deletionTimestamp, drop the finalizer straight away without connecting to ZK. Retrying an unreachable server makes sense on the create path; on the delete path it shouldn't be allowed to block teardown.

Lenguaje dominante
Rust
Estrellas
37
Forks
11
Merge medio
1 d 8 h
PR fusionados (30 d)
10

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de stackabletech/zookeeper-operator

Todos los issues de stackabletech/zookeeper-operator

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.