google / google/stellar-engine

[Bug] 2-networking-a-fedramp-high: NVA return-route generation only routes the FIRST tenant subnet per environment ([0] index) — additional tenant subnets get no NVA return route and lose ALL internet egress

Open Beginner friendly
#146 0 comments 0 reactions 0 assignees View on GitHub
bug Level of Effort - Low Priority - Medium
Dominant language
HCL
Stars
49
Forks
20
Avg merge
3d 14h
Merged PRs (30d)
28

Description

## Bug Description
In `fast/stages-aw/2-networking-a-fedramp-high/nva.tf`, the NVA cloud-config's trusted-side ("landing") return routes are built from a per-environment comprehension that selects only the FIRST tenant subnet of each environment:
```hcl
locals {
routing_config = [
{ name = "dmz", enable_masquerading = true, routes = [var.gcp_ranges.gcp_dmz_primary] },
{ name = "landing", routes = [for k, v in var.envs_folders : module.env-spoke-vpc[k].subnets[
"${var.regions.primary}/${[for s in try(var.subnets[lower(k)], []) : s.name if s.tenant != null][0]}"
].ip_cidr_range] },
]
}
```
The trailing `[…][0]` means each environment contributes exactly ONE tenant subnet CIDR to the NVA's return routes. But `var.subnets` is typed `map(list(object(... name, ip_cidr_range, tenant = optional(string) ...)))` — a LIST of subnets per network — so an environment can legitimately hold multiple tenant subnets. Any tenant subnet that is not the first in its environment's list receives **no return route** on the NVA.

The NVA (the `simple-nva` cloud-config, `enable_masquerading = true` on the DMZ interface) masquerades outbound workload traffic on its untrusted/DMZ interface and forwards it correctly. But when the reply returns, the NVA looks up the original client IP, finds no trusted-side route for that subnet, falls through to its **own default route**, and sends the reply back out the DMZ/untrusted interface toward the internet gateway instead of back to the trusted spoke. The connection hangs. Net effect for VMs in the un-routed subnet: name resolution still works (the metadata resolver needs no egress) but **100% of real internet egress (TCP and ICMP) times out** — a silent, hard-to-diagnose failure.

## Environment and Deployment Context
* **Stellar Engine Version/Commit:** `main` (`fast/stages-aw/2-networking-a-fedramp-high/nva.tf`, `local.routing_config` "landing" routes)
* **Deployment Type:**
* [ ] US Region Restricted (e.g., Access Policy constraint)
* [ ] FedRAMP Medium
* [x] FedRAMP High
* [ ] DoD IL4
* [ ] DoD IL5
* [ ] Stand-alone / Custom
* **FAST Stage (if applicable):**
* [ ] Stage 0 (Bootstrap)
* [ ] Stage 1 (Resource Management)
* [x] Stage 2 (Network Creation)
* [ ] Stage 3 (Security and Audit)
* **Affected Component:** `fast/stages-aw/2-networking-a-fedramp-high/nva.tf` — `local.routing_config`, the `landing` `routes` comprehension (`[for s in try(var.subnets[lower(k)], []) : s.name if s.tenant != null][0]`); the value is consumed by `module.nva-cloud-config` (`modules/cloud-config-container/simple-nva`) and pushed to both NVAs as their trusted-side static routes.

## Steps to Reproduce
1. Deploy an SE FRH landing zone whose prod environment spoke has TWO tenant subnets — e.g. a base tenant subnet (`10.1.0.0/24`) and a second tenant subnet (`10.1.1.0/24`), both with `tenant` set.
2. Bring up a VM in the SECOND subnet (`10.1.1.x`), no external IP.
3. From that VM: `curl -m 15 https://www.google.com` and `ping -c2 8.8.8.8` — both time out; DNS still resolves (metadata resolver at 169.254.169.254).
4. On either NVA: `ip route get 10.1.1.4` → resolves `via dev eth0` (the default/untrusted interface), NOT toward the trusted spoke; `ip route` shows landing return routes only for each environment's FIRST tenant subnet (`10.1.0.0/24`, `10.2.0.0/24`, `10.3.0.0/24`) and none for `10.1.1.0/24`.
5. Add the missing route on both NVAs: `sudo ip route replace 10.1.1.0/24 via dev eth1` — egress immediately works (HTTP 200, ping 0% loss). This confirms the NVA return-route omission is the sole cause.

## Expected Behavior
The NVA is given return routes for ALL tenant subnets in every environment, so every tenant subnet's VMs egress correctly through the NVA + Cloud NAT.

## Actual Behavior
Only the first tenant subnet per environment gets a return route; VMs in any additional tenant subnet lose all internet egress (silently — DNS still resolves), while the NVA misroutes their return traffic out its untrusted interface.

## Relevant Logs and Errors
```
# from a VM in the SECOND tenant subnet (10.1.1.4):
$ curl -sS -m 15 https://www.google.com
curl: (28) Connection timed out after 15000 milliseconds
$ ping -c2 8.8.8.8
2 packets transmitted, 0 received, 100% packet loss

# on the NVA:
$ ip route get 10.1.1.4
10.1.1.4 via 10.0.0.1 dev eth0 src 10.0.0.20 # <- out the DMZ/default iface, WRONG WAY
$ ip route | grep -E '10\.[123]\.'
10.1.0.0/24 via 10.0.1.1 dev eth1
10.2.0.0/24 via 10.0.1.1 dev eth1
10.3.0.0/24 via 10.0.1.1 dev eth1 # <- no route for 10.1.1.0/24
```

## Expected/Suggested Fix
Replace the single-subnet `[0]` pick with a `flatten` over ALL tenant subnets per environment:
```hcl
routes = flatten([for k, v in var.envs_folders : [
for s in try(var.subnets[lower(k)], []) :
module.env-spoke-vpc[k].subnets["${var.regions.primary}/${s.name}"].ip_cidr_range
if s.tenant != null
]])
```
This routes every tenant subnet back through the NVA, so adding a second tenant subnet no longer silently breaks its egress.

## Additional Context
Verified live on an SE FRH deployment: before the fix, egress from the subnet was 100% broken; adding the missing NVA return route on both NVAs restored it (confirmed HTTP 200 + a Cloud NAT public egress IP + ping 0% loss). Belongs with the other subnet-handling gaps in the gemini ingress path (#105 hardcoded LB subnet, #111 fragile network resolution), but this one is upstream in **Stage-2 networking**, not the gemini blueprint. NOTE: the ephemeral `ip route replace` fix does NOT survive an NVA reboot / MIG replacement / networking-stage redeploy — the durable fix is the source change above (or a persistent route baked into the NVA cloud-config).

Contributor guide

Open the contributing guide

Research direction

Start in fast/stages-aw/2-networking-a-fedramp-high/nva.tf at local.routing_config and its landing routes comprehension. Trace how the routes value reaches module.nva-cloud-config and the simple-nva configuration. Done means every tenant subnet in each environment contributes a return-route CIDR, including additional subnets, and the NVA egress scenario no longer loses return traffic.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, terraform
Domain
cloud, infrastructure, networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.