How to configure internet access if VPP uses non-default interface?

Environment

• Calico/VPP version: v0.17.0-calicov3.20.2.
• Kubernetes version: 1.21.1 .
• Deployment type: bare-metal on equinix.metal, host nodes OS is Ubuntu 20.04 LTS.
• Network configuration:
  • master node:
    1. default interface + ip - bond0 + 139.178.65.125.
    2. VPP interface + ip - eno2 + 10.0.0.1/30.
  • worker node:
    1. default interface + ip - bond0 + 139.178.65.127.
    2. VPP interface + ip - eno2 + 10.0.0.2/30.

Issue description
Pods are unable to ping internet when VPP uses non-default interface.

To Reproduce
Steps to reproduce the behavior:

1. Start 2 bare-metal nodes with the following interfaces configured:
   1. bond0 - bond interface, has access to the internet.
   2. eno2 - simple interface, has access to the eno2 on the other node.
2. Setup additional IP addresses:
   1. master node - 10.0.0.1/30 dev eno2.
   2. worker node - 10.0.0.2/30 dev eno2.
3. Create kubernetes cluster.
4. Edit Calico calico-vpp-nohuge.yaml deployment:

  vpp_dataplane_interface: eno2

5. Start Calico with the modified deployment.
6. Run kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml.
7. Run kubectl exec dnsutils -- ping -c 4 8.8.8.8.
8. ping fails.

Expected behavior
dnsutils pod should be able to access internet.

Start with the Calico VPP deployment configuration in calico-vpp-nohuge.yaml and reproduce the failure using the documented bond0/eno2 setup. Check the networking path used when vpp_dataplane_interface is eno2, then verify completion by running kubectl exec dnsutils -- ping -c 4 8.8.8.8 successfully.