Unable to route traffic from LoadBalancer to Secondary Interface

Unable to maintain healthy backend probe checks with two Virtual Machines using two network interface cards where services being balanced are on the secondary interface.

I have a web cluster where each VM has two network interfaces:

- eth0 - management interfac
- eth1 - data interface (web traffic)

The VM's are running a web server listening only on the data interface.

I also have a load balancer situated in front of the VM's and I have the backend pools configured using the data interface (eth1) on the VM's. LB Health probes are checking port 443.

I have replaced the default route/gateway on each VM so that all traffic routes out through the data interface. so my routing table looks like this:

````
default via 172.20.192.1 dev eth1
168.63.129.16 via 172.21.192.1 dev eth0 proto static
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
169.254.169.254 via 172.21.192.1 dev eth0 proto static
172.20.192.0/21 dev eth1 proto kernel scope link src 172.20.192.4
172.21.192.0/21 dev eth0 proto kernel scope link src 172.21.192.4
````

I believe waagent is adding this particular route:

168.63.129.16 via 172.21.192.1 dev eth0 proto static

Load Balancer health probes originate from 168.63.129.16, so the VM's will receive the health probe traffic on the secondary interface but then try to route the response on the primary interface which in my case, is incorrect.

**Additional context**

When I manually remove the static route and add in the correct static route for the traffic over data, the health probe succeeds and the load balancing works.

````
default via 172.20.192.1 dev eth1
168.63.129.16 via 172.20.192.1 dev eth1 proto static
````

However, this configuration will not persist. After restarting the network or rebooting, the route is restored. I believe this is done via the waagent. This is from the waagent log, showing the ExtHandler restoring the routing table

````
2020-06-08T14:49:21.121812Z WARNING EnvHandler ExtHandler Dhcp client is not running.
2020-06-08T14:49:26.206672Z INFO EnvHandler ExtHandler EnvMonitor: Detected dhcp client restart. Restoring routing table.
2020-06-08T14:49:26.217306Z INFO EnvHandler ExtHandler Configure routes
2020-06-08T14:49:26.227839Z INFO EnvHandler ExtHandler Gateway:None
2020-06-08T14:49:26.232420Z INFO EnvHandler ExtHandler Routes:None
````

Azure Image: CIS CentOS Linux 7 Benchmark
Distro: Centos 7.8
WALinuxAgent-2.2.45 running on centos 7.8.2003
Python: 2.7.5
Goal state agent: 2.2.48.1

Note that the static route being added doesn't have a metric, so linux will using 0 which is the highest priority. Therefore, I can't add another custom route with a higher priority. Is it possible to add on a metric to the static route being added for the special Azure IP (168.63.129.16), this way users can another static route with a higher priority.

e.g.
168.63.129.16 via 172.20.192.1 dev eth1 proto static metric 999
168.63.129.16 via 172.21.192.1 dev eth0 proto static metric 1000

Start with the WALinuxAgent ExtHandler path that logs “Configure routes” and restores the route for 168.63.129.16; reproduce the routing-table change after a network restart on the described CentOS setup. Done means the restored route supports the requested metric or the supported configuration is clearly documented, and health probes reach the secondary interface successfully.

Written by the indexing model from the issue text.