Reuse the binary planfile from `atmos terraform plan` in `atmos terraform generate planfile`
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 175
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 134
Description
### Describe the Feature
Add an opt-in `--reuse-plan[=auto|always|never]` flag to `atmos terraform generate planfile` so that, when a canonical binary planfile from a prior `atmos terraform plan` already exists on disk, the JSON/YAML conversion step skips the (expensive) re-plan and runs only `terraform show -json` against the existing binary.
Today `atmos terraform generate planfile` always runs the full pipeline: `terraform init` → `terraform plan -out=` → `terraform show -json ` → delete tmpfile. The first two steps are by far the most expensive (state refresh, provider calls), and they duplicate work that `atmos terraform plan` already performed when the user invoked it earlier in the same workflow.
### Expected Behavior
- `--reuse-plan=never` *(default)* — current behavior. Always run a fresh plan into a temp dir, then `show -json`. Preserves the documented semantic that `generate planfile` produces a fresh plan.
- `--reuse-plan=auto` — reuse the canonical binary planfile (`/-.planfile`) **if and only if** staleness gates pass. Otherwise fall back to a fresh plan and log at `Debug` which gate failed.
- `--reuse-plan=always` — reuse strictly. If no canonical binary exists or any staleness gate fails, return an error. Intended for CI orchestration where a missing prior plan should be a hard failure.
Staleness gates for `auto` / `always`:
1. The canonical binary planfile exists on disk.
2. The binary's mtime is newer than `.terraform.lock.hcl` in the same component dir.
3. The binary's mtime is newer than any `*.tf`, `*.tf.json`, or `*.tfvars*` file under the component workdir.
4. `atmosConfig.Components.Terraform.Plan.SkipPlanfile` is `false` (TFC / `--skip-planfile=true` workflows never produce a binary, so reuse is impossible).
5. The current invocation's `--process-templates` / `--process-functions` / `--skip` flags do not contradict what was used to produce the binary. (If we cannot detect this cheaply, document that the user is responsible.)
### Use Case
1. **CI workflows that combine `plan` and `generate planfile`.** A common pipeline runs `atmos terraform plan -s ` for the standard plan summary/upload, and then `atmos terraform generate planfile -s ` to emit JSON for `checkov`, `conftest`, OPA, or a custom policy gate. Today this re-runs `terraform plan` end-to-end — typically tens of seconds per component on a non-trivial state. With `--reuse-plan=always`, this collapses to a sub-second `terraform show -json` step.
2. **Interactive review.** A user runs `atmos terraform plan` locally, inspects the output, then wants the JSON form for a one-off check. `--reuse-plan=auto` does the right thing without re-refreshing state.
3. **Composes with #2497.** The CI integration tracked in #2497 wires `generate planfile` into the `atmos ci` hook lifecycle. In that flow, the binary is guaranteed fresh (same SHA, same `init`, no human intervention between plan and generate), so reuse is both safe and high-value there.
### Describe Ideal Solution
Implementation surface is small:
- Add the `--reuse-plan` flag in [`cmd/terraform/generate/planfile.go`](https://github.com/cloudposse/atmos/blob/main/cmd/terraform/generate/planfile.go) and thread it into `PlanfileOptions` in [`pkg/terraform/options.go`](https://github.com/cloudposse/atmos/blob/main/pkg/terraform/options.go).
- In [`ExecuteTerraformGeneratePlanfile`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_generate_planfile.go), branch:
- When `reusePlan != never` and the staleness gates pass, call `getTerraformPlanJSON(...canonicalBinaryPath)` directly instead of `generateNewPlanFile(...)`. `getTerraformPlanJSON` already runs `terraform init` + `terraform show -json` (see [`terraform_plan_diff.go:197-218`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_plan_diff.go#L197-L218)).
- When `reusePlan == always` and a gate fails, return a typed error explaining which gate failed.
- The canonical binary path is already exposed via [`constructTerraformComponentPlanfilePath`](https://github.com/cloudposse/atmos/blob/main/internal/exec/path_utils.go#L76-L81), which is the same path `atmos terraform plan` writes via `-out` ([`terraform_execute_helpers_args.go:31`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_execute_helpers_args.go#L31)). So the layouts already align — both commands share the same base name, with JSON/YAML written at `.json`/`.yaml` ([`terraform_generate_planfile.go:237`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_generate_planfile.go#L237)).
The pattern itself is not new — the codebase already treats "stored binary + `terraform show -json`" as a first-class shape:
- [`VerifyPlanfile`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_verify_plan.go#L24-L114) runs `getTerraformPlanJSON` against a stored binary and compares to a freshly generated plan.
- The CI plugin's [`downloadPlanfileForVerification`](https://github.com/cloudposse/atmos/blob/main/pkg/ci/plugins/terraform/handlers.go#L202-L258) downloads a stored binary before `before.terraform.deploy` and expects a sibling fresh plan to be generated for comparison.
`generate planfile` is the odd one out.
### Adjacent cleanups worth bundling
- `getTerraformPlanJSON` runs `runTerraformInit` even when the caller (`ExecuteTerraformGeneratePlanfile`) already initialized via `generateNewPlanFile`. A cheap `initDone` guard at [`terraform_plan_diff.go:198-201`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_plan_diff.go#L198-L201) would save a redundant init on every invocation regardless of reuse.
- The internally-generated binary at `/new.plan` is unconditionally deleted ([`terraform_generate_planfile.go:171-176`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_generate_planfile.go#L171-L176)). When `--reuse-plan=always` is set (or via a separate `--persist-binary` flag), promoting that binary to the canonical path on success would let a subsequent `atmos terraform apply --planfile` / `atmos terraform planfile upload` consume it without yet another re-plan — same delegation pattern `VerifyPlanfile` already uses at [`terraform_verify_plan.go:107-110`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_verify_plan.go#L107-L110).
### Alternatives Considered
1. **Always reuse, no flag.** Rejected because the documented semantic of `generate planfile` is "Atmos executes `terraform plan` … then `terraform show`" ([website/docs/cli/commands/terraform/generate/planfile.mdx:26](https://github.com/cloudposse/atmos/blob/main/website/docs/cli/commands/terraform/generate/planfile.mdx#L26)). Silently swapping in a stale binary would surprise users running it for Checkov/policy gates outside CI, where the binary on disk may be hours or days old.
2. **Add reuse as an env var only (no flag).** Rejected for discoverability. A flag with three explicit values (`never`/`auto`/`always`) signals intent in workflow files and CI logs.
3. **Implicit "reuse if `--ci` is set" coupling.** Rejected — too magical. The reuse semantic is orthogonal to the CI lifecycle work in #2497; users may want one without the other.
4. **Cache by hashing component inputs.** Overkill. The mtime-based staleness gates are cheap, deterministic, and match how Terraform itself decides plan validity at apply time.
### Additional Context
Related:
- #2497 — wire `generate planfile` into the `atmos ci` hook lifecycle. This issue and #2497 are complementary: #2497 adds the hooks; this issue makes the hook payload cheap to produce in CI.
- #1403 — adds `--dir` flag to `generate planfile` (orthogonal — path ergonomics).
- #194 — closed; about reusing a plan for `apply`, not for JSON conversion.
Contributor guide
Research direction
Start in cmd/terraform/generate/planfile.go and pkg/terraform/options.go, then trace ExecuteTerraformGeneratePlanfile in internal/exec/terraform_generate_planfile.go and getTerraformPlanJSON in internal/exec/terraform_plan_diff.go. Verify the reuse modes and staleness gates against constructTerraformComponentPlanfilePath, with focused tests for fresh, reusable, and rejected binaries; done means default behavior remains unchanged and auto/always produce the specified outcomes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, terraform
- Domain
- cli, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100