feat(infra): Preliminary Report: Migrate Classic Application Load Balancers to Global External ALBs

> **Important**: This is a **preliminary report** outlining the roadmap to migrate our existing Classic Application Load Balancers (ALBs) to modern Global External Application Load Balancers (`EXTERNAL_MANAGED`). With the adoption of Google Provider v8.0.0 in #2807, our load balancers remain on Classic ALB (`load_balancing_scheme = "EXTERNAL"`). This issue tracks the planned, staged migration to `EXTERNAL_MANAGED`. An engineer should verify these findings and steps against the live GCP infrastructure before executing any production changes. Useful verification tips and commands are provided at the end of this issue.

---

## Overview

In `hashicorp/google` provider 8.0.0, the default `load_balancing_scheme` was changed to `EXTERNAL_MANAGED` (Envoy-based Global External ALB). To avoid disruptive API errors and maintain stability, #2807 explicitly pinned existing load balancers to `load_balancing_scheme = "EXTERNAL"`.

This issue tracks the follow-up project: migrating our frontend and backend load balancers from Classic ALB (`EXTERNAL`) to Global External ALB (`EXTERNAL_MANAGED`).

Because Google Cloud disallows changing the load balancing scheme directly in place on active infrastructure via Terraform, this requires an orchestrated, multi-step migration using GCP's migration state machine.

---

## Resources Subject to Migration

The following 6 load balancing resources will be migrated across both environments:

### Staging Environment (`webstatus-dev-public-staging`)
- **Backend Services (2)**:
- `staging-frontend-service` (`infra/frontend/service.tf:L148`)
- `staging-backend-service` (`infra/backend/service.tf:L342`)
- **Global Forwarding Rules (4)**:
- `staging-frontend-https-rule` (`infra/frontend/service.tf:L183`)
- `staging-frontend-main-https-rule` (`infra/frontend/service.tf:L197`)
- `staging-backend-https-rule` (`infra/backend/service.tf:L377`)
- `staging-backend-main-https-rule` (`infra/backend/service.tf:L391`)

### Production Environment (`webstatus-dev-public-prod`)
- **Backend Services (2)**:
- `prod-frontend-service`
- `prod-backend-service`
- **Global Forwarding Rules (4)**:
- `prod-frontend-https-rule`
- `prod-frontend-main-https-rule`
- `prod-backend-https-rule`
- `prod-backend-main-https-rule`

---

## Step-by-Step Migration Procedure

Execute on **Staging** first (`PROJECT=webstatus-dev-public-staging`, `ENV=staging`), observe for at least one full release cycle, then execute on **Production** (`PROJECT=webstatus-dev-public-prod`, `ENV=prod`).

### Phase 1: Migrate Backend Services
> Note: In GCP, every backend service attached to a forwarding rule must be migrated to `EXTERNAL_MANAGED` before migrating any forwarding rules.

1. **Set Migration State to `PREPARE`**:
Prepares Envoy proxy infrastructure alongside existing GFEs.
```bash
gcloud compute backend-services update ${ENV}-frontend-service \
--project=${PROJECT} --global --external-managed-migration-state=PREPARE

gcloud compute backend-services update ${ENV}-backend-service \
--project=${PROJECT} --global --external-managed-migration-state=PREPARE
```
*Wait at least 6 minutes for Google edge infrastructure propagation. Verify state reports "Prepared for test".*

2. **Canary Test via `TEST_BY_PERCENTAGE` (Recommended for Prod)**:
Routes a canary percentage through the modern Envoy data plane.
```bash
gcloud compute backend-services update ${ENV}-frontend-service \
--project=${PROJECT} --global --external-managed-migration-state=TEST_BY_PERCENTAGE \
--external-managed-migration-testing-percentage=10

gcloud compute backend-services update ${ENV}-backend-service \
--project=${PROJECT} --global --external-managed-migration-state=TEST_BY_PERCENTAGE \
--external-managed-migration-testing-percentage=10
```
*Wait at least 6 minutes. Monitor error rates, latencies, and proxy logs.*

3. **Transition to `TEST_ALL_TRAFFIC`**:
Routes 100% of traffic to the Envoy-based ALB infrastructure.
```bash
gcloud compute backend-services update ${ENV}-frontend-service \
--project=${PROJECT} --global --external-managed-migration-state=TEST_ALL_TRAFFIC

gcloud compute backend-services update ${ENV}-backend-service \
--project=${PROJECT} --global --external-managed-migration-state=TEST_ALL_TRAFFIC
```
*Wait at least 6 minutes. Verify health checks and traffic stability.*

4. **Finalize Backend Services to `EXTERNAL_MANAGED`**:
```bash
gcloud compute backend-services update ${ENV}-frontend-service \
--project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute backend-services update ${ENV}-backend-service \
--project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED
```
*Wait at least 6 minutes.*

### Phase 2: Migrate Forwarding Rules
Once backend services are `EXTERNAL_MANAGED`:
```bash
gcloud compute forwarding-rules update ${ENV}-frontend-https-rule \
--project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute forwarding-rules update ${ENV}-frontend-main-https-rule \
--project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute forwarding-rules update ${ENV}-backend-https-rule \
--project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute forwarding-rules update ${ENV}-backend-main-https-rule \
--project=${PROJECT} --global --load-balancing-scheme=EXTERNAL_MANAGED
```
*Wait at least 6 minutes.*

### Phase 3: Rollback Procedure (Safety Mechanism)
GCP maintains a 90-day rollback window:
1. Revert forwarding rules first:
`gcloud compute forwarding-rules update --project=${PROJECT} --global --load-balancing-scheme=EXTERNAL`
2. Transition backend services back to `TEST_ALL_TRAFFIC`, wait 6 minutes, then set `--load-balancing-scheme=EXTERNAL`.

### Phase 4: Terraform Code Synchronization
After both environments are migrated:
1. Update `infra/frontend/service.tf` and `infra/backend/service.tf` to set `load_balancing_scheme = "EXTERNAL_MANAGED"` (or remove the explicit setting since it is the default in provider v8).
2. Run `terraform plan` inside the DevContainer; verify zero infrastructure drift.

---

## Tips for Verification & Next Steps

1. **Check Live Schemes via `gcloud`**:
```bash
gcloud compute backend-services describe staging-backend-service --project=webstatus-dev-public-staging --global --format="value(loadBalancingScheme)"
gcloud compute forwarding-rules describe staging-backend-https-rule --project=webstatus-dev-public-staging --global --format="value(loadBalancingScheme)"
```
2. **Monitor HTTP Edge Metrics in Cloud Monitoring**:
During `TEST_BY_PERCENTAGE` and `TEST_ALL_TRAFFIC`, monitor Cloud Monitoring metric `loadbalancing.googleapis.com/https/request_count` grouped by `response_code_class` to verify zero unexpected 5xx responses.
3. **Reference Documentation**:
- [Google Cloud: Migrate resources from Classic to Global External Application Load Balancer](https://cloud.google.com/load-balancing/docs/https/migrate-to-global-external-alb)

Start by reading infra/frontend/service.tf and infra/backend/service.tf, then verify the listed staging backend services and forwarding rules with the provided gcloud describe commands. Follow the documented staging migration phases before production, monitor Cloud Monitoring request counts and 5xx responses, and finish by setting the Terraform schemes and confirming zero drift with terraform plan in the DevContainer.

Written by the indexing model from the issue text.