Interconnect: Add private DNS zones with domain forwarding/peering to enable Private Google Access or Private Service Connect workload calls from on-prem - using AWS VPN for Interconnect simulation

In addition to interconnect work in #298 and #318
Shadow/port: https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/286
see perimeter work done in #446
Also, for DNS forwarding return - I thought I read that the 34.199.192.0/19 range was implicitly advertised but hidden on all BGP routing - it turns out it is up to the customer to advertise this - by adding a custom route directly on the interconnect/vpn router beside the dynamic BGP routes - found this a bit odd but I can see optional case for those not wanting cloud to ground and only ground to cloud for SCED/SC2G
- Key for cloud to ground private DNS forwarded queries - add the network-patterns https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3152 slide 27 referenced ingress proxy 34.199.192.0/19 as a custom route on the interconnect router (in addition to dynamic subnet routes) - without this DNS queries from gcp to on-prem will not resolve.
- specifically - https://cloud.google.com/dns/docs/policies#return-route
"For Cloud VPN tunnels that use dynamic routing or for Cloud Interconnect, [configure a custom route advertisement](https://cloud.google.com/network-connectivity/docs/router/how-to/advertising-custom-ip) for 35.199.192.0/19 on the BGP session of the Cloud Router that manages the tunnel or VLAN attachment."
- https://cloud.google.com/network-connectivity/docs/router/how-to/advertising-custom-ip

# architecture
- Options for interconnect sharing are less secure when using peering to the VPC that owns the interconnects - but is lower cost
-

# Design Issues
## DI 0: Decide on network architecture
- we are option B Hybrid Connectivity to Multiple VPC (or Shared VPC) networks
- slide 27 - B3: Hybrid DNS Option 1
- slide 30 - B6: Private Service Connect (PSC) for Google APIs
https://docs.google.com/presentation/d/13sjT2tJ4yLIYGRREE3wBrylB1OvcEMpKdquVuJB_nX4/edit?resourcekey=0-N3DruQaiutFvZ98HTT7-vQ#slide=id.g1154b3b950f_2_3611
-

-

found older notes I made in March in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/318
## DI 1: DNS Forwarding (type 2 to on prem dns server) and DNS Peering
- https://cloud.google.com/dns/docs/zones/forwarding-zones#console
- https://cloud.google.com/dns/docs/zones/peering-zones
- https://cloud.google.com/dns/docs/zones/zones-overview#forwarding-peering-example
- https://cloud.google.com/blog/products/networking/how-to-use-cloud-dns-peering-in-a-shared-vpc-environment

# simulation
- 3 VPCs host, perimeter, on-prem (all need private google access set)
- vpc peering between host and perimeter
- private zones in on-prem site.local and perimeter cloud.local
- dns peering in host to
- dns forwarding from on-prem to host
- dns forwarding from host to on-prem

# Testing
Using existing VPN in ol for PSC simulation (however global routing across 2 regions is set - may need a regional VPN)

on the VMs get dig
```
sudo apt-get install dnsutils -y
dig www.google.com
```

## Procedures
- https://cloud.google.com/dns/docs/policies#create-in

## Implementation
- on prem DNS servers are authoritative for on prem zones
- cloud DNS is authoritative for GCP zones
- DNS queries from GCP to on-prem use 35.199.192.0/19
### Manual
- simulate on-prem...(other CSP or eBGP enabled home router)
- create dns-host-ol project - host project enabled on shared vpc creation
- enable apis
```
dns.googleapis.com
```
- set permissions
Required 'compute.organizations.enableXpnHost' permission for 'projects/dns-host-ol' - https://gcp.permissions.cloud/predefinedroles/compute.xpnAdmin
- create custom vpc (for reuse as shared vpc)
```
gcloud compute networks create host --project=dns-host-ol --description=hody --subnet-mode=custom --mtu=1460 --bgp-routing-mode=regional
gcloud compute networks subnets create host-sn --project=dns-host-ol --description=host-sn --range=10.0.2.0/24 --stack-type=IPV4_ONLY --network=host --region=northamerica-northeast1 --enable-private-ip-google-access
```
- create host shared-vpc (custom networks, dns api enabled)
-
- create dns private zones in host project
```
gcloud dns --project=dns-host-ol managed-zones create private --description="" --dns-name="private.obrienlabs.dev." --visibility="private" --networks="host"
```
- enable compute api
- create shared vpc
- create service project dns-sp-ol
- create dns egress proxy in shared vpc
- create inbound server policy in Cloud DNS for on prem
- create forwarding zone in Cloud DNS for on prem
-
- On Prem DNS
-
-

### Automated