cloudposse / cloudposse/atmos

Reuse the binary planfile from `atmos terraform plan` in `atmos terraform generate planfile`

Open
#2,498 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.4k
Forks
175
Avg merge
2d 3h
Merged PRs (30d)
134

Description

### Describe the Feature

Add an opt-in `--reuse-plan[=auto|always|never]` flag to `atmos terraform generate planfile` so that, when a canonical binary planfile from a prior `atmos terraform plan` already exists on disk, the JSON/YAML conversion step skips the (expensive) re-plan and runs only `terraform show -json` against the existing binary.

Today `atmos terraform generate planfile` always runs the full pipeline: `terraform init` → `terraform plan -out=` → `terraform show -json ` → delete tmpfile. The first two steps are by far the most expensive (state refresh, provider calls), and they duplicate work that `atmos terraform plan` already performed when the user invoked it earlier in the same workflow.

### Expected Behavior

- `--reuse-plan=never` *(default)* — current behavior. Always run a fresh plan into a temp dir, then `show -json`. Preserves the documented semantic that `generate planfile` produces a fresh plan.
- `--reuse-plan=auto` — reuse the canonical binary planfile (`/-.planfile`) **if and only if** staleness gates pass. Otherwise fall back to a fresh plan and log at `Debug` which gate failed.
- `--reuse-plan=always` — reuse strictly. If no canonical binary exists or any staleness gate fails, return an error. Intended for CI orchestration where a missing prior plan should be a hard failure.

Staleness gates for `auto` / `always`:

1. The canonical binary planfile exists on disk.
2. The binary's mtime is newer than `.terraform.lock.hcl` in the same component dir.
3. The binary's mtime is newer than any `*.tf`, `*.tf.json`, or `*.tfvars*` file under the component workdir.
4. `atmosConfig.Components.Terraform.Plan.SkipPlanfile` is `false` (TFC / `--skip-planfile=true` workflows never produce a binary, so reuse is impossible).
5. The current invocation's `--process-templates` / `--process-functions` / `--skip` flags do not contradict what was used to produce the binary. (If we cannot detect this cheaply, document that the user is responsible.)

### Use Case

1. **CI workflows that combine `plan` and `generate planfile`.** A common pipeline runs `atmos terraform plan -s ` for the standard plan summary/upload, and then `atmos terraform generate planfile -s ` to emit JSON for `checkov`, `conftest`, OPA, or a custom policy gate. Today this re-runs `terraform plan` end-to-end — typically tens of seconds per component on a non-trivial state. With `--reuse-plan=always`, this collapses to a sub-second `terraform show -json` step.

2. **Interactive review.** A user runs `atmos terraform plan` locally, inspects the output, then wants the JSON form for a one-off check. `--reuse-plan=auto` does the right thing without re-refreshing state.

3. **Composes with #2497.** The CI integration tracked in #2497 wires `generate planfile` into the `atmos ci` hook lifecycle. In that flow, the binary is guaranteed fresh (same SHA, same `init`, no human intervention between plan and generate), so reuse is both safe and high-value there.

### Describe Ideal Solution

Implementation surface is small:

- Add the `--reuse-plan` flag in [`cmd/terraform/generate/planfile.go`](https://github.com/cloudposse/atmos/blob/main/cmd/terraform/generate/planfile.go) and thread it into `PlanfileOptions` in [`pkg/terraform/options.go`](https://github.com/cloudposse/atmos/blob/main/pkg/terraform/options.go).
- In [`ExecuteTerraformGeneratePlanfile`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_generate_planfile.go), branch:
- When `reusePlan != never` and the staleness gates pass, call `getTerraformPlanJSON(...canonicalBinaryPath)` directly instead of `generateNewPlanFile(...)`. `getTerraformPlanJSON` already runs `terraform init` + `terraform show -json` (see [`terraform_plan_diff.go:197-218`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_plan_diff.go#L197-L218)).
- When `reusePlan == always` and a gate fails, return a typed error explaining which gate failed.
- The canonical binary path is already exposed via [`constructTerraformComponentPlanfilePath`](https://github.com/cloudposse/atmos/blob/main/internal/exec/path_utils.go#L76-L81), which is the same path `atmos terraform plan` writes via `-out` ([`terraform_execute_helpers_args.go:31`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_execute_helpers_args.go#L31)). So the layouts already align — both commands share the same base name, with JSON/YAML written at `.json`/`.yaml` ([`terraform_generate_planfile.go:237`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_generate_planfile.go#L237)).

The pattern itself is not new — the codebase already treats "stored binary + `terraform show -json`" as a first-class shape:

- [`VerifyPlanfile`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_verify_plan.go#L24-L114) runs `getTerraformPlanJSON` against a stored binary and compares to a freshly generated plan.
- The CI plugin's [`downloadPlanfileForVerification`](https://github.com/cloudposse/atmos/blob/main/pkg/ci/plugins/terraform/handlers.go#L202-L258) downloads a stored binary before `before.terraform.deploy` and expects a sibling fresh plan to be generated for comparison.

`generate planfile` is the odd one out.

### Adjacent cleanups worth bundling

- `getTerraformPlanJSON` runs `runTerraformInit` even when the caller (`ExecuteTerraformGeneratePlanfile`) already initialized via `generateNewPlanFile`. A cheap `initDone` guard at [`terraform_plan_diff.go:198-201`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_plan_diff.go#L198-L201) would save a redundant init on every invocation regardless of reuse.
- The internally-generated binary at `/new.plan` is unconditionally deleted ([`terraform_generate_planfile.go:171-176`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_generate_planfile.go#L171-L176)). When `--reuse-plan=always` is set (or via a separate `--persist-binary` flag), promoting that binary to the canonical path on success would let a subsequent `atmos terraform apply --planfile` / `atmos terraform planfile upload` consume it without yet another re-plan — same delegation pattern `VerifyPlanfile` already uses at [`terraform_verify_plan.go:107-110`](https://github.com/cloudposse/atmos/blob/main/internal/exec/terraform_verify_plan.go#L107-L110).

### Alternatives Considered

1. **Always reuse, no flag.** Rejected because the documented semantic of `generate planfile` is "Atmos executes `terraform plan` … then `terraform show`" ([website/docs/cli/commands/terraform/generate/planfile.mdx:26](https://github.com/cloudposse/atmos/blob/main/website/docs/cli/commands/terraform/generate/planfile.mdx#L26)). Silently swapping in a stale binary would surprise users running it for Checkov/policy gates outside CI, where the binary on disk may be hours or days old.
2. **Add reuse as an env var only (no flag).** Rejected for discoverability. A flag with three explicit values (`never`/`auto`/`always`) signals intent in workflow files and CI logs.
3. **Implicit "reuse if `--ci` is set" coupling.** Rejected — too magical. The reuse semantic is orthogonal to the CI lifecycle work in #2497; users may want one without the other.
4. **Cache by hashing component inputs.** Overkill. The mtime-based staleness gates are cheap, deterministic, and match how Terraform itself decides plan validity at apply time.

### Additional Context

Related:

- #2497 — wire `generate planfile` into the `atmos ci` hook lifecycle. This issue and #2497 are complementary: #2497 adds the hooks; this issue makes the hook payload cheap to produce in CI.
- #1403 — adds `--dir` flag to `generate planfile` (orthogonal — path ergonomics).
- #194 — closed; about reusing a plan for `apply`, not for JSON conversion.

Contributor guide

Open the contributing guide

Research direction

Start in cmd/terraform/generate/planfile.go and pkg/terraform/options.go, then trace ExecuteTerraformGeneratePlanfile in internal/exec/terraform_generate_planfile.go and getTerraformPlanJSON in internal/exec/terraform_plan_diff.go. Verify the reuse modes and staleness gates against constructTerraformComponentPlanfilePath, with focused tests for fresh, reusable, and rejected binaries; done means default behavior remains unchanged and auto/always produce the specified outcomes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, terraform
Domain
cli, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.