drt: terraform infrastructure, migrate to private addresses, and add soft GC alerts

## Summary

Terraform the `cockroach-drt` GCE infrastructure, make DRT-provisioned GCE clusters private by default, migrate existing clusters off public addressing, and introduce non-destructive TTL monitoring for expired DRT clusters.

Part of #77644. Related private-address work: #173114, #173117, #173118, and cockroachlabs/cockroach#3302.

## Motivation

DRT currently differs from the newly terraformed `crl-e2e-infra` environment:

- `cockroach-drt` still uses the GCE `default` network and permissive legacy firewall rules.
- DRT VMs receive public addresses.
- DRT uses the default Compute Engine service account.
- Connectivity, NAT, routing, firewall, and regional subnet infrastructure are not managed consistently through Terraform.
- Existing `drtprod` scripts and configurations contain public-address assumptions.
- Private GCE managed instance groups cannot currently create load balancers; that work is tracked by #173118.

DRT is also intentionally excluded from destructive scheduled GC because its clusters are long-lived. This creates a foot gun: engineers sometimes create temporary smoke-test clusters in `cockroach-drt`, assume the configured TTL will cause automatic cleanup, and leave the cluster behind.

DRT clusters must remain protected from automatic deletion, but expired TTLs should produce actionable Slack warnings.

## Infrastructure work

Terraform the required `cockroach-drt` infrastructure using the same security model as `crl-e2e-infra`:

- Create a custom `cockroach-drt-vpc`.
- Create regional subnets using the roachprod naming convention: `${project}-vpc-${region}`.
- Cover all regions used by active DRT configurations:
- `us-east1`
- `us-central1`
- `us-east4`
- `us-east5`
- `northamerica-northeast2`
- Enable Private Google Access.
- Configure Cloud NAT and regional routers for outbound access.
- Add proxy-only subnets where required by the internal load-balancer architecture from #173118.
- Attach the VPC to the required NCC/private-routing infrastructure so it is reachable from engineer, TeamCity, TestEng, monitoring, and workload environments.
- Replace unrestricted ingress with explicit internal, IAP, corporate, TeamCity, health-check, and monitoring rules plus deny-all ingress.
- Create and use a dedicated least-privilege `roachprod-vm@cockroach-drt` service account.
- Preserve and document the intentional project split unless deliberately changed:
- compute, infrastructure, and metadata: `cockroach-drt`
- DNS: `cockroach-shared`
- shared artifacts: `crl-e2e-infra`
- Import or replace existing manually managed resources as appropriate.
- Remove the legacy default network and permissive firewall rules after migration.

## DRT tooling changes

Update both DRT entrypoints and active YAML configurations:

- Make GCE creates resolve to private mode, preferably through `--address-mode=auto` so AWS DRT configurations remain public and supported.
- Decide whether normal DRT access uses direct private routing, IAP, or both, and configure `--gce-use-iap` where required.
- Update the Go `drtprod` binary and legacy `scripts/drtprod` wrapper consistently.
- Ensure predefined and ad hoc DRT GCE creates cannot silently retain public addressing.
- Fix `scripts/drtprod dns`, which currently uses `adminurl --ips` and therefore requires public addresses.
- Publish private node addresses in `drt.crdb.io` and remove stale public A records.
- Validate SSH, run, put/get, logs, SQL, Admin UI, monitoring, Datadog, GCS access, and cluster-to-cluster communication.
- Resolve or gate the PUA configurations that call `load-balancer create` until #173118 supports private MIG load balancers.

## Existing-cluster migration

Inventory every provisioned DRT cluster and create a per-cluster migration runbook.

At investigation time, the primary `drt-1m-tables` cluster consists of three zonal MIGs and uses Local SSDs. GCP does not support moving MIG members between VPCs in place, so it requires new private instance templates/groups and a data-preserving migration.

For each cluster:

- Determine whether it can be recreated, restored from backup, or migrated through replication.
- Back up or otherwise protect Local SSD data before stopping or replacing VMs.
- Create the replacement cluster without public addresses.
- Validate health, workloads, monitoring, DNS, and operational commands.
- Cut over workload clients.
- Remove the public cluster and stale DNS records.
- Document rollback steps.

## Soft GC for DRT

Run scheduled TTL detection against `cockroach-drt`, but never automatically delete DRT resources.

The protection should be structural rather than relying solely on someone remembering `--dry-run`:

- Add a `notify-only` or equivalent project GC policy for `cockroach-drt`.
- Ensure the DRT GC path cannot delete clusters, VMs, disks, load balancers, or DNS records.
- Preserve explicit manual deletion through `drtprod destroy`.
- Detect every cluster whose TTL has expired, including canonical long-running DRT clusters.
- Post an actionable warning to the agreed DRT Slack channel and, when possible, notify the cluster owner.
- Include:
- cluster name and owner
- expiration time and how long it has been expired
- project and cloud
- suggested `drtprod extend` and `drtprod destroy` commands
- Deduplicate repeated notifications while retaining an appropriate reminder cadence until the cluster is extended or removed.
- Fail closed when inventory is incomplete or cloud access fails.
- Monitor the scheduled job itself so failures are visible.

Existing GC status classification and Slack notification code may be reused, but notify-only messages must say "expired" rather than incorrectly claiming that resources were destroyed.

## Acceptance criteria

- [ ] `cockroach-drt` infrastructure is managed by Terraform.
- [ ] All required DRT regions have private subnets, NAT, routing, and appropriate firewall coverage.
- [ ] No unrestricted ingress firewall rule remains.
- [ ] New DRT GCE clusters receive no public addresses by default.
- [ ] AWS and mixed-cloud DRT workflows retain supported behavior.
- [ ] Private create, sync, start, SSH, file transfer, logs, SQL, DNS, monitoring, and destroy workflows pass.
- [ ] Private MIG load-balancer workflows are supported or clearly gated on #173118.
- [ ] Existing provisioned DRT clusters have been migrated and validated.
- [ ] No DRT VM or private load balancer exposes a public frontend.
- [ ] Expired DRT TTLs generate actionable Slack warnings.
- [ ] Scheduled DRT GC cannot delete any cloud or DNS resource.
- [ ] Manual `drtprod destroy` remains available.
- [ ] Rollback and operational runbooks are documented.

## Out of scope

- Automatically deleting expired DRT clusters.
- Changing address defaults for non-DRT projects.
- Making AWS DRT clusters private.
- Replacing the internal-load-balancer implementation tracked by #173118.

Epic: #77644

Jira issue: CRDB-66636

Start by comparing the existing cockroach-drt setup with the terraformed crl-e2e-infra environment, then inspect the Go drtprod entrypoint, legacy scripts/drtprod wrapper, active YAML configurations, and existing GC status and Slack notification code. Review the related issues before choosing migration and private-load-balancer approaches. Done means the listed Terraform, private-address, migration, workflow, and notify-only GC acceptance criteria pass.

Written by the indexing model from the issue text.