network: L3 routed networking without shared L2 domain

### The required feature described as a wish

# Networking without Layer 2
This proposal is to add a new networking feature to CloudStack where Instances are directly assigned /32 (IPv4) and/or /128 (IPv6) addresses without a shared Layer 2 domain.

A shared Layer 2 domain in this case would be a VLAN or VXLAN VNI where Instances shared the same Broadcast/Multicast domain and where they use a shared IP-gateway for their routing.

# Layer 3
By leveraging various features of the Linux kernel, making this a KVM-only feature, we can directly route an IPv4 and/or IPv6 address to a virtual machine by using a dynamic routing protocol like BGP, but this could also work with OSPF(v3).

By eliminating the need for Layer 2 we can create a routed network where no Instance has a "network relationship" with another Instance. Every Instance has one or more routes installed in the routing table of the network and can be routed to any host at any time.

In the examples below I will use two IP-addresses:
- 2.57.57.30
- 2001:678:3a4:100::80

# Hypervisor host as gateway
On the hypervisor the cloudbr0 bridge will be created and assigned an IPv4 and IPv6 address:

```
auto cloudbr0
iface cloudbr0 inet static
address 169.254.0.1/32
address fe80::1/64
bridge-ports none
bridge-stp off
bridge-fd 0
```

All Instances will be connected to this bridge and they will be configured to use the following IP-gateways:
- 169.254.0.1
- fe80::1

# Inside the Instance
As there is no Layer 2 available the IP-configuration within the VM has to be done using ConfigDrive for cloud-init, a Virtual Router handing out DHCP and cloud-init data is not possible in this design. For the VM there is no way of detecting the cloud-init source over the network as our current CloudStack provider within cloud-init relies on the DHCP server as a source.

After the Instance has used cloud-init to fetch the networking information from ConfigDrive the Netplan (Ubuntu Linux) configuration would look like this:

```
network:
ethernets:
ens18:
accept-ra: no
nameservers:
addresses:
- 2620:fe::fe
- 2620:fe::9
addresses:
- 2.57.57.30/32
- 2001:678:3a4:100::80/128
routes:
- to: default
via: fe80::1
- to: default
via: 169.254.0.1
on-link: true
version: 2
```

In this configuration the network inside the Instance is configured to the the addresses configured on **cloudbr0** as the gateway, meaning that the hypervisor will act as the gateway and route the IP-traffic.

This results in the interface being configured:

```
2: ens18: mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:02:45:76:d2:35 brd ff:ff:ff:ff:ff:ff
altname enp0s18
inet 2.57.57.30/32 scope global ens18
valid_lft forever preferred_lft forever
inet6 2001:678:3a4:100::80/128 scope global
valid_lft forever preferred_lft forever
```

All hypervisors will use an identical configuration for cloudbr0, this allows all Instances to have the same routes in their route table:

```
root@web01:~# ip -6 route show
::1 dev lo proto kernel metric 256 pref medium
2001:678:3a4:100::80 dev ens18 proto kernel metric 256 pref medium
fe80::/64 dev ens18 proto kernel metric 256 pref medium
default via fe80::1 dev ens18 proto static metric 1024 pref medium
root@web01:~# ip -4 route show
default via 169.254.0.1 dev ens18 proto static onlink
root@web01:~#
```

# ARP and NDP neighbor configuration
CloudStack is aware of the IPv4 and/or IPv6 addresses assigned to an Instance as well as the MAC address. On the hypervisor these entries have to be installed into the kernel's routing table and neighbor table. In this example the commands would be:

```
ip -6 route add 2001:678:3a4:100::80/128 dev cloudbr0
ip -6 neigh add 2001:678:3a4:100::80 lladdr 52:02:45:76:d2:35 dev cloudbr0 nud permanent
ip -4 route add 2.57.57.30/32 dev cloudbr0
ip -4 neigh add 2.57.57.30 lladdr 52:02:45:76:d2:35 dev cloudbr0 nud permanent
```

These entries would need to be added upon Instance start on that host and removed on Instance stop/migrate. The KVM Agent should handle the orchestration of these entries.

# Dynamic Routing
Configuring these entries in the routing table is not sufficient, these need to be advertised to the upstream network. For this the hypervisor host would need to use some form of dynamic routing. BGP is the most commonly used, while others would like to use OSPF(v3).

In both cases the hypervisor will announce these /32 (IPv4) and /128 (IPv6) addresses to the upstream network while receiving a default route (0.0.0.0/0 and ::/0) from the network to be able to route traffic.

A very simple piece of configuration for FRRouting (BGP or OSPF) could be:

## BGP
```
router bgp
redistribute kernel route-map only-cloud
!
route-map only-cloud permit 10
match interface cloudbr0
```

## OSPF
```router ospf
redistribute kernel route-map only-cloud
network YOUR_NETWORK/XX area 0.0.0.0
!
route-map only-cloud permit 10
match interface cloudbr0
```

# IP address pools
As each Instance is assigned a IPv4 and/or IPv6 address there is no need to create a "network" inside CloudStack. The concept would be that CloudStack simply has a pool of addresses to choose from and allocates them to an Instance

A pool could be:

- 2.57.57.80
- 145.31.53.21
- 90.78.37.15
- 88.17.11.53

- 2001:db8::100
- 2001:678:3a4:100::80
- 2a00:f10:415:27::100

These addresses have no relationship with eachother, but they don't have to as each individual address is assigned the a VM

This networking setup also allows for very easy single stack IPv6-only Virtual Machines where IPv4 can be added or removed when needed. There is no dependency on either of the two protocols.

# Summary
This networking design completely eliminates the use of Layer 2 broadcast/multicast domains. Each Instance becomes a full L3 routed part of the network where CloudStack's orchestration will make sure the addresses are routed towards the host where the Instance is on.

Using this setup it's very easy to create a massively scalable and reliable network spanning multiple datacenters as there is no shared L2 or VXLAN overlay.

The most common use-case for this feature will probably be public cloud providers which need to assign public IPv4/IPv6 addresses to Instance and want to share nothing between the VMs.

首先追蹤 KVM Agent 的網路協調，以及現有的 ConfigDrive 和 cloud-init 整合。檢視目前如何處理 cloudbr0、執行個體位址配置，以及主機層級的路由和鄰居項目，然後找出涵蓋 VM 啟動、停止和遷移的測試。完成的定義是針對路由的 /32 和 /128 位址、上游路由廣告，以及生命週期清理，制定明確的實作計畫。

由索引模型根據 Issue 內容生成。