google / google/stellar-engine
[Bug] 2-networking-a-fedramp-high: NVA return-route generation only routes the FIRST tenant subnet per environment ([0] index) — additional tenant subnets get no NVA return route and lose ALL internet egress
- Dominant language
- HCL
- Stars
- 49
- Forks
- 20
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 28
Description
## Bug Description
In `fast/stages-aw/2-networking-a-fedramp-high/nva.tf`, the NVA cloud-config's trusted-side ("landing") return routes are built from a per-environment comprehension that selects only the FIRST tenant subnet of each environment:
```hcl
locals {
routing_config = [
{ name = "dmz", enable_masquerading = true, routes = [var.gcp_ranges.gcp_dmz_primary] },
{ name = "landing", routes = [for k, v in var.envs_folders : module.env-spoke-vpc[k].subnets[
"${var.regions.primary}/${[for s in try(var.subnets[lower(k)], []) : s.name if s.tenant != null][0]}"
].ip_cidr_range] },
]
}
```
The trailing `[…][0]` means each environment contributes exactly ONE tenant subnet CIDR to the NVA's return routes. But `var.subnets` is typed `map(list(object(... name, ip_cidr_range, tenant = optional(string) ...)))` — a LIST of subnets per network — so an environment can legitimately hold multiple tenant subnets. Any tenant subnet that is not the first in its environment's list receives **no return route** on the NVA.
The NVA (the `simple-nva` cloud-config, `enable_masquerading = true` on the DMZ interface) masquerades outbound workload traffic on its untrusted/DMZ interface and forwards it correctly. But when the reply returns, the NVA looks up the original client IP, finds no trusted-side route for that subnet, falls through to its **own default route**, and sends the reply back out the DMZ/untrusted interface toward the internet gateway instead of back to the trusted spoke. The connection hangs. Net effect for VMs in the un-routed subnet: name resolution still works (the metadata resolver needs no egress) but **100% of real internet egress (TCP and ICMP) times out** — a silent, hard-to-diagnose failure.
## Environment and Deployment Context
* **Stellar Engine Version/Commit:** `main` (`fast/stages-aw/2-networking-a-fedramp-high/nva.tf`, `local.routing_config` "landing" routes)
* **Deployment Type:**
* [ ] US Region Restricted (e.g., Access Policy constraint)
* [ ] FedRAMP Medium
* [x] FedRAMP High
* [ ] DoD IL4
* [ ] DoD IL5
* [ ] Stand-alone / Custom
* **FAST Stage (if applicable):**
* [ ] Stage 0 (Bootstrap)
* [ ] Stage 1 (Resource Management)
* [x] Stage 2 (Network Creation)
* [ ] Stage 3 (Security and Audit)
* **Affected Component:** `fast/stages-aw/2-networking-a-fedramp-high/nva.tf` — `local.routing_config`, the `landing` `routes` comprehension (`[for s in try(var.subnets[lower(k)], []) : s.name if s.tenant != null][0]`); the value is consumed by `module.nva-cloud-config` (`modules/cloud-config-container/simple-nva`) and pushed to both NVAs as their trusted-side static routes.
## Steps to Reproduce
1. Deploy an SE FRH landing zone whose prod environment spoke has TWO tenant subnets — e.g. a base tenant subnet (`10.1.0.0/24`) and a second tenant subnet (`10.1.1.0/24`), both with `tenant` set.
2. Bring up a VM in the SECOND subnet (`10.1.1.x`), no external IP.
3. From that VM: `curl -m 15 https://www.google.com` and `ping -c2 8.8.8.8` — both time out; DNS still resolves (metadata resolver at 169.254.169.254).
4. On either NVA: `ip route get 10.1.1.4` → resolves `via dev eth0` (the default/untrusted interface), NOT toward the trusted spoke; `ip route` shows landing return routes only for each environment's FIRST tenant subnet (`10.1.0.0/24`, `10.2.0.0/24`, `10.3.0.0/24`) and none for `10.1.1.0/24`.
5. Add the missing route on both NVAs: `sudo ip route replace 10.1.1.0/24 via dev eth1` — egress immediately works (HTTP 200, ping 0% loss). This confirms the NVA return-route omission is the sole cause.
## Expected Behavior
The NVA is given return routes for ALL tenant subnets in every environment, so every tenant subnet's VMs egress correctly through the NVA + Cloud NAT.
## Actual Behavior
Only the first tenant subnet per environment gets a return route; VMs in any additional tenant subnet lose all internet egress (silently — DNS still resolves), while the NVA misroutes their return traffic out its untrusted interface.
## Relevant Logs and Errors
```
# from a VM in the SECOND tenant subnet (10.1.1.4):
$ curl -sS -m 15 https://www.google.com
curl: (28) Connection timed out after 15000 milliseconds
$ ping -c2 8.8.8.8
2 packets transmitted, 0 received, 100% packet loss
# on the NVA:
$ ip route get 10.1.1.4
10.1.1.4 via 10.0.0.1 dev eth0 src 10.0.0.20 # <- out the DMZ/default iface, WRONG WAY
$ ip route | grep -E '10\.[123]\.'
10.1.0.0/24 via 10.0.1.1 dev eth1
10.2.0.0/24 via 10.0.1.1 dev eth1
10.3.0.0/24 via 10.0.1.1 dev eth1 # <- no route for 10.1.1.0/24
```
## Expected/Suggested Fix
Replace the single-subnet `[0]` pick with a `flatten` over ALL tenant subnets per environment:
```hcl
routes = flatten([for k, v in var.envs_folders : [
for s in try(var.subnets[lower(k)], []) :
module.env-spoke-vpc[k].subnets["${var.regions.primary}/${s.name}"].ip_cidr_range
if s.tenant != null
]])
```
This routes every tenant subnet back through the NVA, so adding a second tenant subnet no longer silently breaks its egress.
## Additional Context
Verified live on an SE FRH deployment: before the fix, egress from the subnet was 100% broken; adding the missing NVA return route on both NVAs restored it (confirmed HTTP 200 + a Cloud NAT public egress IP + ping 0% loss). Belongs with the other subnet-handling gaps in the gemini ingress path (#105 hardcoded LB subnet, #111 fragile network resolution), but this one is upstream in **Stage-2 networking**, not the gemini blueprint. NOTE: the ephemeral `ip route replace` fix does NOT survive an NVA reboot / MIG replacement / networking-stage redeploy — the durable fix is the source change above (or a persistent route baked into the NVA cloud-config).
Contributor guide
Research direction
Start in fast/stages-aw/2-networking-a-fedramp-high/nva.tf at local.routing_config and its landing routes comprehension. Trace how the routes value reaches module.nva-cloud-config and the simple-nva configuration. Done means every tenant subnet in each environment contributes a return-route CIDR, including additional subnets, and the NVA egress scenario no longer loses return traffic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, terraform
- Domain
- cloud, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100