Discussing immutable infrastructure principle

TamasZsiros commented in https://github.com/cncf/telecom-user-group/pull/29/files#r379228510
> One thing to maybe keep in mind is that while the microservices configuration is not changed throughout their lifecycle, a telecom application (built up of many microservices) might need to expose configuration on a higher abstraction level. For example you might want to change the general behavior of a CNF in the network (represented via a configuration management object exposed vie a nort-bound interface to a management system). This setting does not directly translate to "microservice configuration", still needs to be accessed, stored, managed, and still determimes the behavior of the CNF.

@taylor responded with:

> Yes, I can see higher-level abstractions of configuration and management as something that could be desired in larger applications. That is possible while following cloud native principles (including composability in this case).
>
> If I'm understanding correctly, then it seems like the behaviour can be managed by the external management system directly with the CNFs cloud native API or through a "configuration management" service which communicates with the CNFs and provides a higher level of abstraction for the management system. That configuration management service should itself be cloud native of course for the entire "larger application" to be cloud native.
>
> The CNF modifiable run-time behaviour should be adjusted via APIs. State for restarts, scaling, etc, should be stored outside of the CNF (eg. adjust max sessions to 10k).

@jeffsaelens chimed in with:
> in the immutable infra principles, Watson and I have started to distinguish between Day 0 and Day n types of configuration changes (L5). Your assumptions are correct Taylor. The deployment and makeup of the service itself (microservices with versioning, dependencies, operators, etc...) should be handled as laid out. Day n config modifications to the CNF itself (add VLAN ## to Port #/#) have to allow mutability. We aren't going to nuke a CNF router every time we want to add a VLAN or Route, nor are we going to blue/green some rolling deployment. That being said, tying back in to the earlier sections declaring the need for orchestration, this Day n configuration should still be handled by all the principles expressed in the rest of the paper. They should be stored in a version controlled template and pushed in an atomic manner via orchestration.

@dabernie said:

> In my view, and I tend to agree with the "immutable infra principles", a CNF deployment state (ie interfaces, mem, libraries, device drivers, version, etc.) cannot be modified at runtime. How it behaves when a packet enters into it could be totally reprogrammable. For example, I could run a CNF Data Plane with P4Runtime which makes its forwarding pipeline totally modifiable but which does not change its nature to the underlying infra.