Redesign idea: "Central nervous system for Kured"

Hello everyone.

I guess this was supposed to be a blog post, but I will make it an issue instead. We can discuss it better here.

## Intro
Our lovely kured works well for many. It's simple, and I would like to keep it that way.

Yet, we recently had more and more special cases, which requires building a bit more complexity around our code.
Slowly, we'll increase our complexity, and reduce our readability.

Next to this, we have seen PRs and issues in the past mentioning shortcomings of our current architecture, for flexibility [1] [2] or security reasons [3].

I would like us to think about what "another kured" (for more complex cases) could be.
Here are my thoughts...

## Separate kured daemonset in components

Instead of having a single daemonset doing everything, I would separate in multiple components:
- An operator **brain**, analysing the cluster state, and orchestrating the **Node Reboot Procedure**(s)
- A **Node Reboot Detector**, a user-defined daemonset, which will probably hold one or multiple containers.

### The Brain

The brain would be based on the current code from `rebootAsRequired`.
Instead of locking on the kured daemonset, it would lock on a separate object, like a config map.
The rest would be like the current model: checking for any reboot blockers, draining the node (which requires patch node permissions), and triggering a reboot of one to multiple nodes (depending on the **"Nodes Maintenance Window"**). Because the reboot wouldn't be local anymore, the brain would create a *job* to trigger a reboot scheduled on the necessary node, job which can be described in the **"Reboot Procedure"**.

Configuration of the behaviour of the brain would be done by editing custom resources and/or config maps.

### The "Node reboot detector"

This daemonset will detect if the node needs rebooting, and simultaneously report it back to the "brain" and optionally expose that metric to prometheus. The latter is similar to our `maintainRebootRequiredMetric`. We can have a consistent period check between the *brain* and the *detector* by making sure both of them read `period` from the same config map/CR.
Alternatively to "report it back to the brain", the brain could `fetch` the node information exposed with the `rebootRequiredMetric`, or the `reboot detector` could `PATCH` a node with the relevant information (maybe by relying on Node Admission controller by mounting kubelet's credentials). TBD.

The daemonset is likely to be a single container, which will check the presence of a file at an expected location `/kured/reboot-required`. This will only require read access to that volume. No need for `privileged` or `hostPIDs`.

For the basic ubuntu workloads, this is the same as having our daemonset, with a volume mapping `/var/run` to `/kured/` in read-only mode.

For custom workloads, one user could define it's own daemonset containers, additional to our "standard one". The standard container would still check the presence of the file `/kured/reboot-required. However, in the custom case, that file could be mounted from a volume that's reachable from another container in the daemonset. That other container would implement its the own logic to detect if a reboot is required.

## What do we gain?

### Flexibility
The node detector structure allows the users to implement their own detection systems without any code change from kured.
The brain architecture allows the users to implement their own *reboot procedures* without any code change from kured, and would allow more flexible maintenance windows.

### Simplicity
The detector is now simpler, as it is just watching over a single file, on a read-only file system, with no extra privileges.
The brain simplifies some code, as the coordination features like throttling will be easier to implement.
However, we will have a more complex architecture with an operator.

### Security
We can isolate components better, so we can tighten the access controls.

The `detector` daemonset would only need to access to:
- the `brain` config map, in read only, to know the check period
- (optionally) a volume to mount from host, in read-only mode.
- a write access to a shared object (cm/cr) to expose the state of the node (or not, based on the above alternative choice).

The `brain` would need access to:
- A custom lock object (improving security, see also [4])
- The necessary access to drain/cordon/uncordon nodes
- The necessary access to create jobs with the necessary rights based on user input (those reboot procedure might be privileged access!). I recently had success with a reboot on a systemd host using only the hostPID privileges!

Note: With the reboot procedure _outside_ the kured code, we can think of using less privileged access: Using an API to reboot a node, using different reboot methods. This would also allow to use seccomp profiles in multiple places!

### Scaling operations

By adding a NodesMaintenanceWindow CRD, a user can define multiple maintenances, including exceptional ones, by adding a new "Maintenance Window". This will be read by the `brain` when necessary. Creating a Custom resource will allow us to create new fields, like "Serialization", which would define how many nodes are to be rebooted at the same time for example.

[1]: https://github.com/weaveworks/kured/issues/358
[2]: https://github.com/weaveworks/kured/issues/314
[3]: https://github.com/weaveworks/kured/issues/25
[4]: https://github.com/weaveworks/kured/issues/60