hackforla / hackforla/devops

Migrate the Terraform S3 backends off dynamodb_table onto use_lockfile and delete the lock tables

Aperta
#218 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
complexity: medium feature: maintenance role: DevOps Engineer size: 2pt
Lingua principale
PowerShell
Stelle
8
Fork
10
Merge medio
7h 30m
PR unite (30g)
22

Descrizione

### Overview

We need to replace the deprecated `dynamodb_table` backend parameter with `use_lockfile = true` in `hackforla/incubator` and `hackforla/devops-security`, then delete the two now-unused DynamoDB lock tables, because HashiCorp deprecated DynamoDB-based state locking in Terraform 1.11 and will remove it in a future minor version. Both repos already run 1.16.1, so every plan and apply in both is printing the deprecation warning today.

### Action Items

**Facts verified 2026-09-07, so you do not have to re-derive them:**

- The warning fires 3x per plan run and 4x per apply, in every run sampled in both repos back to 2026-08-08. It reads `Warning: Deprecated Parameter / The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile" instead.` and points at `backend.tf line 7`, which is misleading — that line is an empty `backend "s3" {}` block. The parameter is in `terraform/prod.backend.tfvars`, which the workflows pass as `backend_config_file`.
- `use_lockfile` went generally available in Terraform 1.11, in the same release that deprecated the DynamoDB arguments. Both repos resolve `required_version = "~> 1.16.0"` to **1.16.1**, so there is no version work.
- **No new IAM permission is needed.** S3-native locking needs `s3:GetObject`, `s3:PutObject` and `s3:DeleteObject` on `.tflock`. The apply roles (`incubator-tf-apply`, `devops-security-tf-apply`) hold `AdministratorAccess`. The plan roles never take a lock at all — `dflook/terraform-plan` passes `-lock=false` unconditionally (`image/entrypoints/plan.sh:15`, checked at both the pinned `v1.49.0` and at `main`). That is also why the plan roles work today holding only `ReadOnlyAccess` over DynamoDB.
- Both state buckets (`hfla-incubator-terraform-state`, `hfla-ops-terraform-state`) have versioning **Enabled**.
- Both lock tables hold exactly **one** item and it is not a lock — it is the `…/terraform.tfstate-md5` digest row. No lock is currently held in either.
- **Both tables have `DeletionProtectionEnabled: true`**, so deleting them is a two-call operation. Neither has point-in-time recovery.
- These are the **only two DynamoDB tables in account `035866691871`**. After step 3 the account has none.

---

**1. hackforla/incubator** — 1 PR, `terraform/prod.backend.tfvars`:

- [ ] Delete the `dynamodb_table = "hfla_incubator_terraform_table"` line and add `use_lockfile = true`.
- [ ] **Add a no-op comment line to any `.tf` file in the same PR.** Both workflows filter on `paths: ['**/*.tf', '**/*.yaml']`, and `prod.backend.tfvars` matches neither pattern — so a PR that changes only the tfvars triggers **no workflow at all**, on the PR or on the merge. This is not optional: without it there is nothing to verify against and the merge produces no apply run.
- [ ] **Treat a non-empty plan as a hard gate.** A backend change produces zero resource changes. If the plan shows any, stop and raise it rather than merging.
- [ ] After the PR merges, confirm the `Terraform apply (OIDC)` run succeeds and that **no** `Deprecated Parameter` warning appears anywhere in its log.

**2. hackforla/devops-security** — 1 PR. This one is bigger than incubator's, because three other things in this repo reference the lock table:

- [ ] `terraform/prod.backend.tfvars` — delete `dynamodb_table = "hfla_ops_terraform_table"`, add `use_lockfile = true`.
- [ ] `terraform/aws-custom-policies/tf-plan-scoped.json` — delete the whole `AllowDynamoDBLockTableAccessForTerraformPlan` statement. This file is **live**: `terraform/aws-custom-policies.tf` feeds it to `modules/aws-policies`, which deploys it as the IAM policy `IncubatorTfPlanSecretsRead`, currently attached to the `incubator-tf-plan` role. Leaving the statement would leave a live policy granting DynamoDB actions on a table that no longer exists.
- **Do not add `.tflock` permissions to replace it.** The statement immediately above it already grants `s3:PutObject`/`s3:DeleteObject` on `arn:aws:s3:::hfla-ops-terraform-state/*`, which covers the lock file — and the plan role does not lock anyway.
- **Do not "fix" the fact that a policy named `IncubatorTfPlanSecretsRead`, attached to incubator's plan role, grants access to devops-security's backend.** That cross-wiring predates this work. If it looks wrong to you, raise it as its own issue.
- [ ] `CONTRIBUTING.md` — delete the `#### Set up DynamoDB to store the backend state` section (lines 157–174, through the `***` after `Back to Top`), and in the **Creating Local tfvars file** example (~line 251) swap `dynamodb_table = "hfla_ops_terraform_table"` for `use_lockfile = true`. This is the section that tells every new member to create a lock table by hand.
- [ ] `.github/ISSUE_TEMPLATE/pre-work-template-devops-security.md` — delete the `- [ ] Create the DynamoDB table` sub-item (line 40) and reword its parent (line 38) so it no longer promises a DynamoDB step.
- [ ] Same no-op `.tf` comment requirement, and it is worse here: this repo's filter is `paths: ['**/*.tf']` only, so `.tfvars`, `.json`, `.md` and the issue template all fail to trigger it.
- [ ] **The plan gate is different in this repo.** Expect **exactly one** change: an in-place update of `module.aws_custom_policies.aws_iam_policy.custom_policy["IncubatorTfPlanSecretsRead"]`, from the JSON edit. Anything else — any replacement, any second resource — is a stop-and-raise.
- [ ] After the PR merges, confirm `Apply Terraform changes on merge` succeeds with no `Deprecated Parameter` warning, and that the policy's new default version no longer contains the DynamoDB statement.
- [ ] **After the merge, re-open `/issues/new/choose` and confirm the pre-work template renders without the DynamoDB step.** GitHub renders issue templates from the default branch only, so this genuinely cannot be checked from the branch.

**3. Delete both tables** — only after both PRs have merged and both apply runs are green:

- [ ] Confirm no lock is held in either table. `aws dynamodb scan --table-name --region us-west-2 --query 'Items[].LockID.S'` must return only the `-md5` row.
- [ ] Record both table definitions in a comment on this issue **before** deleting, so the change is reversible without a PR trail to read. Capture `aws dynamodb describe-table` and `aws dynamodb list-tags-of-resource` for each.
- [ ] Disable deletion protection, then delete. Both calls need admin in `035866691871` — if you do not have it, hand this step off rather than working around it.

```bash
for t in hfla_incubator_terraform_table hfla_ops_terraform_table; do
aws dynamodb update-table --table-name "$t" --region us-west-2 --no-deletion-protection-enabled
aws dynamodb delete-table --table-name "$t" --region us-west-2
done
```

- [ ] Verify: `aws dynamodb list-tables --region us-west-2` returns an empty list, and the next apply in each repo still succeeds.
- [ ] **To reverse**, recreate the table and re-enable protection. Terraform rewrites the `-md5` digest row itself on the next apply, so the row does not need restoring. `hfla_ops_terraform_table` additionally carried tags `managed-by=exempt` and `project=devops-security`; `hfla_incubator_terraform_table` carried none.

```bash
aws dynamodb create-table --table-name --region us-west-2 \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
aws dynamodb update-table --table-name --region us-west-2 --deletion-protection-enabled
```

**Do the backend swap in one step, not a two-phase migration — but know why:**

- [ ] Terraform documents that `dynamodb_table` and `use_lockfile` may be set **simultaneously**, and that is the official migration path. Its only purpose is to keep two clients that disagree about the locking mechanism from both acquiring a lock. Since holding both keeps the deprecation warning firing, a phased migration means four PRs to clear a warning that two will clear.
- [ ] The risk it covers is real but small here: while this is in flight, a `terraform apply` from a **local checkout that still has `dynamodb_table`** would not see a CI lock taken through S3, or vice versa. Mitigate by not running a local apply against either repo on the day the PRs merge, and by the lock check in step 3.
- [ ] Note for anyone with an existing local checkout: CI initialises from a clean container every run, so there is no cached backend config and no migration prompt. A local `.terraform/` directory **will** prompt `Backend configuration changed` and needs `terraform init -reconfigure`.

**Out of scope — do not do these here:**

- [ ] Do not touch the action pins in these repos' workflows. `actions/checkout`, `aws-actions/configure-aws-credentials` and `dflook/terraform-*` belong to hackforla/devops#183, hackforla/incubator#158, hackforla/incubator#159 and hackforla/devops-security#170.
- [ ] Do not tag or otherwise adopt the two state **buckets**. `hfla-incubator-terraform-state` is still untagged and still shows as unmanaged in the Terraform coverage report; that is separate work.

### Resources/Instructions

- Files, all on `main` in both repos. Line numbers were accurate 2026-09-07 and may drift — locate each by its content rather than by position.
- `hackforla/incubator`: `terraform/prod.backend.tfvars`
- `hackforla/devops-security`: `terraform/prod.backend.tfvars`, `terraform/aws-custom-policies/tf-plan-scoped.json`, `CONTRIBUTING.md`, `.github/ISSUE_TEMPLATE/pre-work-template-devops-security.md`
- [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) — the `use_lockfile` argument, the statement that DynamoDB locking "is deprecated and will be removed in a future minor version", the simultaneous-configuration note, and the `.tflock` permission list.
- [Terraform 1.11 CHANGELOG](https://github.com/hashicorp/terraform/blob/v1.11.0/CHANGELOG.md) — "S3 native state locking is now generally available", the release that introduced the deprecation.
- `dflook/terraform-plan`'s `-lock=false` behaviour: [image/entrypoints/plan.sh](https://github.com/dflook/terraform-github-actions/blob/v1.49.0/image/entrypoints/plan.sh) line 15.
- Runs showing the warning, for comparison after the fix: [incubator apply 34061802778](https://github.com/hackforla/incubator/actions/runs/34061802778), [devops-security apply 34009820416](https://github.com/hackforla/devops-security/actions/runs/34009820416).

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Start with the listed backend files in hackforla/incubator and hackforla/devops-security, then review tf-plan-scoped.json, CONTRIBUTING.md, and the pre-work issue template for all lock-table references. Run the repository Terraform plans and inspect the referenced apply workflows, stopping if changes differ from the stated expectations. Done means both applies are warning-free, the documentation and policy are updated, the template renders correctly, and both DynamoDB tables are recorded, deleted, and verified absent.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, terraform
Ambito
ci-cd, cloud, databases, devops, documentation, infrastructure
Tipo di issue
Refactoring
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Attiva
Chiarezza
Specificata chiaramente
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.