digitalocean / digitalocean/netbox-ip-controller

netbox-ip-controller is not resilient to data loss/corruption in k8s

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
32
Forks
3
PR merge metrics
No merged PRs in 30d

Description

In a scenario that something happens to the underlying storage in k8s, e.g. etcd data is corrupted and has to be restored from a backup, netbox-ip-controller cannot fully recover without manual intervention. The IPs created in NetBox that correspond to the netboxip objects that magically disappeared after data loss, are never deleted, leading to duplicated IPs in NetBox.

A possible approach for solving this:
Tag all IPs created by the controller with some key that is _specific to this particular controller_, for example, k8s cluster name. This will allow netbox-ip-controller to list all of the IPs in NetBox that it created and should be managing. Then add a periodic sync loop which lists all current IPs in NetBox, and makes sure each of them has a parent netboxip object. This sync doesn't need to be running often, as it should only be needed in cases of etcd data loss. The tag that ties IPs to the given controller in netbox needs to be separate from the `--pod-ip-tags/--service-ip-tags` since it has a special meaning and cannot be used for any IPs other than the ones managed by the given controller.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.