Migrate the Terraform S3 backends off dynamodb_table onto use_lockfile and delete the lock tables
- Ngôn ngữ chính
- PowerShell
- Star
- 8
- Fork
- 10
- Merge trung bình
- 7 giờ 30 phút
- Pull request đã merge (30 ngày)
- 22
Mô tả
### Overview
We need to replace the deprecated `dynamodb_table` backend parameter with `use_lockfile = true` in `hackforla/incubator` and `hackforla/devops-security`, then delete the two now-unused DynamoDB lock tables, because HashiCorp deprecated DynamoDB-based state locking in Terraform 1.11 and will remove it in a future minor version. Both repos already run 1.16.1, so every plan and apply in both is printing the deprecation warning today.
### Action Items
**Facts verified 2026-09-07, so you do not have to re-derive them:**
- The warning fires 3x per plan run and 4x per apply, in every run sampled in both repos back to 2026-08-08. It reads `Warning: Deprecated Parameter / The parameter "dynamodb_table" is deprecated. Use parameter "use_lockfile" instead.` and points at `backend.tf line 7`, which is misleading — that line is an empty `backend "s3" {}` block. The parameter is in `terraform/prod.backend.tfvars`, which the workflows pass as `backend_config_file`.
- `use_lockfile` went generally available in Terraform 1.11, in the same release that deprecated the DynamoDB arguments. Both repos resolve `required_version = "~> 1.16.0"` to **1.16.1**, so there is no version work.
- **No new IAM permission is needed.** S3-native locking needs `s3:GetObject`, `s3:PutObject` and `s3:DeleteObject` on `.tflock`. The apply roles (`incubator-tf-apply`, `devops-security-tf-apply`) hold `AdministratorAccess`. The plan roles never take a lock at all — `dflook/terraform-plan` passes `-lock=false` unconditionally (`image/entrypoints/plan.sh:15`, checked at both the pinned `v1.49.0` and at `main`). That is also why the plan roles work today holding only `ReadOnlyAccess` over DynamoDB.
- Both state buckets (`hfla-incubator-terraform-state`, `hfla-ops-terraform-state`) have versioning **Enabled**.
- Both lock tables hold exactly **one** item and it is not a lock — it is the `…/terraform.tfstate-md5` digest row. No lock is currently held in either.
- **Both tables have `DeletionProtectionEnabled: true`**, so deleting them is a two-call operation. Neither has point-in-time recovery.
- These are the **only two DynamoDB tables in account `035866691871`**. After step 3 the account has none.
---
**1. hackforla/incubator** — 1 PR, `terraform/prod.backend.tfvars`:
- [ ] Delete the `dynamodb_table = "hfla_incubator_terraform_table"` line and add `use_lockfile = true`.
- [ ] **Add a no-op comment line to any `.tf` file in the same PR.** Both workflows filter on `paths: ['**/*.tf', '**/*.yaml']`, and `prod.backend.tfvars` matches neither pattern — so a PR that changes only the tfvars triggers **no workflow at all**, on the PR or on the merge. This is not optional: without it there is nothing to verify against and the merge produces no apply run.
- [ ] **Treat a non-empty plan as a hard gate.** A backend change produces zero resource changes. If the plan shows any, stop and raise it rather than merging.
- [ ] After the PR merges, confirm the `Terraform apply (OIDC)` run succeeds and that **no** `Deprecated Parameter` warning appears anywhere in its log.
**2. hackforla/devops-security** — 1 PR. This one is bigger than incubator's, because three other things in this repo reference the lock table:
- [ ] `terraform/prod.backend.tfvars` — delete `dynamodb_table = "hfla_ops_terraform_table"`, add `use_lockfile = true`.
- [ ] `terraform/aws-custom-policies/tf-plan-scoped.json` — delete the whole `AllowDynamoDBLockTableAccessForTerraformPlan` statement. This file is **live**: `terraform/aws-custom-policies.tf` feeds it to `modules/aws-policies`, which deploys it as the IAM policy `IncubatorTfPlanSecretsRead`, currently attached to the `incubator-tf-plan` role. Leaving the statement would leave a live policy granting DynamoDB actions on a table that no longer exists.
- **Do not add `.tflock` permissions to replace it.** The statement immediately above it already grants `s3:PutObject`/`s3:DeleteObject` on `arn:aws:s3:::hfla-ops-terraform-state/*`, which covers the lock file — and the plan role does not lock anyway.
- **Do not "fix" the fact that a policy named `IncubatorTfPlanSecretsRead`, attached to incubator's plan role, grants access to devops-security's backend.** That cross-wiring predates this work. If it looks wrong to you, raise it as its own issue.
- [ ] `CONTRIBUTING.md` — delete the `#### Set up DynamoDB to store the backend state` section (lines 157–174, through the `***` after `Back to Top`), and in the **Creating Local tfvars file** example (~line 251) swap `dynamodb_table = "hfla_ops_terraform_table"` for `use_lockfile = true`. This is the section that tells every new member to create a lock table by hand.
- [ ] `.github/ISSUE_TEMPLATE/pre-work-template-devops-security.md` — delete the `- [ ] Create the DynamoDB table` sub-item (line 40) and reword its parent (line 38) so it no longer promises a DynamoDB step.
- [ ] Same no-op `.tf` comment requirement, and it is worse here: this repo's filter is `paths: ['**/*.tf']` only, so `.tfvars`, `.json`, `.md` and the issue template all fail to trigger it.
- [ ] **The plan gate is different in this repo.** Expect **exactly one** change: an in-place update of `module.aws_custom_policies.aws_iam_policy.custom_policy["IncubatorTfPlanSecretsRead"]`, from the JSON edit. Anything else — any replacement, any second resource — is a stop-and-raise.
- [ ] After the PR merges, confirm `Apply Terraform changes on merge` succeeds with no `Deprecated Parameter` warning, and that the policy's new default version no longer contains the DynamoDB statement.
- [ ] **After the merge, re-open `/issues/new/choose` and confirm the pre-work template renders without the DynamoDB step.** GitHub renders issue templates from the default branch only, so this genuinely cannot be checked from the branch.
**3. Delete both tables** — only after both PRs have merged and both apply runs are green:
- [ ] Confirm no lock is held in either table. `aws dynamodb scan --table-name --region us-west-2 --query 'Items[].LockID.S'` must return only the `-md5` row.
- [ ] Record both table definitions in a comment on this issue **before** deleting, so the change is reversible without a PR trail to read. Capture `aws dynamodb describe-table` and `aws dynamodb list-tags-of-resource` for each.
- [ ] Disable deletion protection, then delete. Both calls need admin in `035866691871` — if you do not have it, hand this step off rather than working around it.
```bash
for t in hfla_incubator_terraform_table hfla_ops_terraform_table; do
aws dynamodb update-table --table-name "$t" --region us-west-2 --no-deletion-protection-enabled
aws dynamodb delete-table --table-name "$t" --region us-west-2
done
```
- [ ] Verify: `aws dynamodb list-tables --region us-west-2` returns an empty list, and the next apply in each repo still succeeds.
- [ ] **To reverse**, recreate the table and re-enable protection. Terraform rewrites the `-md5` digest row itself on the next apply, so the row does not need restoring. `hfla_ops_terraform_table` additionally carried tags `managed-by=exempt` and `project=devops-security`; `hfla_incubator_terraform_table` carried none.
```bash
aws dynamodb create-table --table-name --region us-west-2 \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
aws dynamodb update-table --table-name --region us-west-2 --deletion-protection-enabled
```
**Do the backend swap in one step, not a two-phase migration — but know why:**
- [ ] Terraform documents that `dynamodb_table` and `use_lockfile` may be set **simultaneously**, and that is the official migration path. Its only purpose is to keep two clients that disagree about the locking mechanism from both acquiring a lock. Since holding both keeps the deprecation warning firing, a phased migration means four PRs to clear a warning that two will clear.
- [ ] The risk it covers is real but small here: while this is in flight, a `terraform apply` from a **local checkout that still has `dynamodb_table`** would not see a CI lock taken through S3, or vice versa. Mitigate by not running a local apply against either repo on the day the PRs merge, and by the lock check in step 3.
- [ ] Note for anyone with an existing local checkout: CI initialises from a clean container every run, so there is no cached backend config and no migration prompt. A local `.terraform/` directory **will** prompt `Backend configuration changed` and needs `terraform init -reconfigure`.
**Out of scope — do not do these here:**
- [ ] Do not touch the action pins in these repos' workflows. `actions/checkout`, `aws-actions/configure-aws-credentials` and `dflook/terraform-*` belong to hackforla/devops#183, hackforla/incubator#158, hackforla/incubator#159 and hackforla/devops-security#170.
- [ ] Do not tag or otherwise adopt the two state **buckets**. `hfla-incubator-terraform-state` is still untagged and still shows as unmanaged in the Terraform coverage report; that is separate work.
### Resources/Instructions
- Files, all on `main` in both repos. Line numbers were accurate 2026-09-07 and may drift — locate each by its content rather than by position.
- `hackforla/incubator`: `terraform/prod.backend.tfvars`
- `hackforla/devops-security`: `terraform/prod.backend.tfvars`, `terraform/aws-custom-policies/tf-plan-scoped.json`, `CONTRIBUTING.md`, `.github/ISSUE_TEMPLATE/pre-work-template-devops-security.md`
- [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3) — the `use_lockfile` argument, the statement that DynamoDB locking "is deprecated and will be removed in a future minor version", the simultaneous-configuration note, and the `.tflock` permission list.
- [Terraform 1.11 CHANGELOG](https://github.com/hashicorp/terraform/blob/v1.11.0/CHANGELOG.md) — "S3 native state locking is now generally available", the release that introduced the deprecation.
- `dflook/terraform-plan`'s `-lock=false` behaviour: [image/entrypoints/plan.sh](https://github.com/dflook/terraform-github-actions/blob/v1.49.0/image/entrypoints/plan.sh) line 15.
- Runs showing the warning, for comparison after the fix: [incubator apply 34061802778](https://github.com/hackforla/incubator/actions/runs/34061802778), [devops-security apply 34009820416](https://github.com/hackforla/devops-security/actions/runs/34009820416).
Hướng dẫn đóng góp
Hướng nghiên cứu
Start with the listed backend files in hackforla/incubator and hackforla/devops-security, then review tf-plan-scoped.json, CONTRIBUTING.md, and the pre-work issue template for all lock-table references. Run the repository Terraform plans and inspect the referenced apply workflows, stopping if changes differ from the stated expectations. Done means both applies are warning-free, the documentation and policy are updated, the template renders correctly, and both DynamoDB tables are recorded, deleted, and verified absent.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws, terraform
- Lĩnh vực
- ci-cd, cloud, databases, devops, documentation, infrastructure
- Loại issue
- Tái cấu trúc
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 45/100