google / google/stellar-engine
[Bug] 2-networking-a-fedramp: the NVA cluster runs on Spot VMs with automatic_restart disabled, so a preemption removes the organization's only internet egress path — the IL5 stage runs the equivalent appliances on-demand
- Dominant language
- HCL
- Stars
- 49
- Forks
- 20
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 28
Description
## Bug Description
The FedRAMP High / Moderate networking stage provisions its Network Virtual Appliance cluster on **Spot** instances:
* `nva.tf:83-84` — the `nva-template` `options` block sets `spot = true` and `termination_action = "STOP"` (machine type `n2d-standard-2`, `nva.tf:57`).
* Through `modules/compute-vm/main.tf:294-298`, that resolves on the instance template to `provisioning_model = "SPOT"`, `preemptible = true`, `instance_termination_action = "STOP"` and — because the module computes `automatic_restart = !var.options.spot` — **`automatic_restart = false`**.
* `nva.tf:107-118` — `nva-mig`, `target_size = 2`, autohealing on a TCP:22 health check with `initial_delay_sec = 30`.
* `variables.tf:161-168` — `regions` is typed `object({ primary = string })`, so the `for_each = var.regions` on the template and the MIG (`nva.tf:52,108`) resolves to exactly one element. The cluster is two Spot instances in one regional MIG in **every** deployment, not only small ones.
Those two instances are the organization's only route to the internet, **and that stays true however large the deployment gets**. Workload subnets are free-form in region — `subnets` is typed `map(list(object({ ... region = string ... })))` (`variables.tf:240-262`) — but every part of the egress path is pinned to the single primary region: the NVA template, the NVA MIG and both internal load balancers key off `var.regions` (`nva.tf:52,108,127,156`), and Cloud NAT is a single `dmz-nat-primary` at `var.regions.primary` (`net-vdss.tf:122-130`). Both ILB forwarding rules set `global_access = true` (`nva.tf:135,164`), so subnets outside the primary region reach the primary ILB and hairpin their egress cross-region into those same two appliances.
`nva.tf:184-190` then creates `default-route-nva` — `dest_range = "0.0.0.0/0"`, `next_hop_ilb` pointing at the primary region's vdss ILB — so a multi-region deployment and a single-region one have the identical choke point. While the NVAs are down the route still resolves to the ILB and traffic blackholes: no internet egress anywhere in the organization, in any region, and no inspected path. The larger the deployment, the more it is concentrated on two reclaimable instances.
Spot capacity is reclaimed on 30 seconds' notice whenever Compute Engine needs it back, and reclamation is driven by regional capacity pressure rather than by anything per-instance — so the two instances are not the independent failure domains that `target_size = 2` suggests. With `termination_action = "STOP"` and `automatic_restart = false`, a preempted NVA stays stopped; recovery depends entirely on MIG autohealing repairing it, and that repair has to obtain Spot capacity too. If the region is short of it, the cluster cannot come back, and the outage lasts as long as the shortage does.
The IL5 option makes the opposite choice for the same role — `ngfw.tf:203-206`:
```
scheduling {
automatic_restart = true
on_host_maintenance = "MIGRATE"
}
```
On-demand, restarted automatically, live-migrated on host maintenance. Its MIG also rolls updates with `max_unavailable = 0` and `min_ready_sec = 300` (`ngfw.tf:288-296`) — that stage treats the appliance tier as something that must not lose capacity, while the FedRAMP High stage puts the same tier on instances that can be reclaimed at any moment.
Nothing in the repository records the choice. Neither the stage README nor `docs/tdd.md` mentions Spot, and `git log -S 'spot' -- fast/stages-aw/2-networking-a-fedramp-high/nva.tf` returns a single commit — an unrelated upgrade of the project module — so the setting has never been introduced or changed by a commit that explains it. An operator reading the stage has no way to tell whether it is deliberate.
## Environment and Deployment Context
* **Stellar Engine Version/Commit:** `v2.13.0` (`8f5b67a6`); unchanged at `v3.0.0` (`f64ce6cd`), where the directory is renamed `2-networking-a-fedramp` and the same `spot = true` / `termination_action = "STOP"` pair sits at `nva.tf:83-84`. Because `v3.0.0` folds FedRAMP Moderate into this stage, the Spot NVA cluster is now the shipped pattern for both regimes.
* **Deployment Type:**
* [ ] US Region Restricted (e.g., Access Policy constraint)
* [ ] FedRAMP Medium
* [x] FedRAMP High
* [x] FedRAMP Moderate
* [ ] DoD IL4
* [ ] DoD IL5
* [ ] Stand-alone / Custom
* **FAST Stage (if applicable):** Stage 2 (Networking) — the `a-fedramp` option
* [ ] Stage 0 (Bootstrap)
* [ ] Stage 1 (Resource Management)
* [x] Stage 2 (Networking)
* [ ] Stage 3 (Security)
## Steps to Reproduce
1. Deploy `fast/stages-aw/2-networking-a-fedramp` (FedRAMP High or FedRAMP Moderate).
2. Read the provisioning model off the running appliances:
`gcloud compute instances list --project -net-vdss-host --filter='name~nva' --format='table(name,zone,status,scheduling.provisioningModel,scheduling.instanceTerminationAction,scheduling.automaticRestart)'`
Every NVA reports `SPOT` / `STOP` / `False`.
3. Run the same command against a `2-networking-b-il5-ngfw` deployment for comparison: its appliances report `STANDARD` / `True`.
4. To observe the impact without waiting for a preemption, stop both NVA instances and attempt any outbound connection from a spoke VM. DNS still resolves (the metadata resolver needs no egress); all real egress times out until the instances are running again.
## Expected Behavior
The appliance tier that carries the organization's only egress and inspection path runs on standard (on-demand) instances by default, as the IL5 option's does — or, if Spot is a deliberate cost decision, it is exposed as a stage variable that defaults to standard, and the stage README states the availability trade-off so an operator can make the choice knowingly.
## Actual Behavior
`spot = true` is hardcoded in the instance template with no variable and no documentation, and the derived `automatic_restart = false` means a preempted appliance is not brought back by Compute Engine at all. Verified in a built FedRAMP High organization: the NVA instances report `provisioningModel: SPOT`, and the organization's billing for 2026-08-14 → 2026-08-20 charges them under the Spot N2D core and RAM SKUs.
## Relevant Logs and Errors
None. The apply succeeds and emits no warning — which is the substance of the report: nothing in the plan, the apply output or the stage documentation surfaces that the egress path is preemptible.
## Additional Context
We have not observed a preemption-caused outage in our own deployment; this is reported from the configuration and the failure mode it implies, not from an incident. That is deliberate — the point is that the exposure exists by default and is invisible to whoever inherits it.
**Two further multi-region observations, recorded here rather than filed separately.** First, the NVA's trusted-side return routes are generated from a hardcoded `"${var.regions.primary}/..."` subnet lookup (`nva.tf:30-32`), so a workload subnet outside the primary region receives no return route on the appliance — the failure mode reported in [#146](https://github.com/google/stellar-engine/issues/146), along its region dimension rather than its subnet-index one. Second, the `for_each = var.regions` idiom reads as though written for a multi-region map: were the `regions` type widened to add a `secondary`, the NVA template, the MIG and both ILBs would fan out on their own, but `dmz-nat-primary` and `google_compute_route.default` are pinned to `var.regions.primary` and would not follow — the second region's appliances would be built and never used.
Related: the same file's KMS gap is [#193](https://github.com/google/stellar-engine/issues/193). Both are cases where the FedRAMP High networking stage and the IL5 stage diverge on the appliance tier without the divergence being recorded anywhere.
Contributor guide
Research direction
Start with fast/stages-aw/2-networking-a-fedramp-high/nva.tf:83-84 and trace the options through modules/compute-vm/main.tf:294-298; compare the appliance scheduling in ngfw.tf:203-206. Use the provided gcloud instance listing and a Terraform plan to verify the current behavior. Done means the egress appliance availability choice is safe by default or explicitly configurable, and the stage documentation records the trade-off.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, terraform
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100