cloudposse / cloudposse/atmos

`!terraform.state` in locals is re-evaluated N times per `list stacks` (hang / slowness)

Open
#2,344 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
1.4k
Forks
175
Avg merge
2d 3h
Merged PRs (30d)
134

Description

### Describe the Bug

## What's broken

A single `locals:` block with a few `!terraform.state` refs causes each ref to be evaluated **hundreds or thousands of times** during `atmos list stacks`. Debug log from one run on a ~20-file stack tree with one stack file containing 7 state refs in `locals:`:

```
$ ATMOS_LOGS_LEVEL=Debug atmos list stacks 2>&1 | grep -c 'function="!terraform.state'
1362
```

That's ~195× per ref. Without cloud auth each GCS/S3 call fails fast (~15 ms), so 1362 × 15 ms ≈ 20 s and the command looks hung. With auth, it's still N× slower than necessary.

## Minimal repro

```
repro/
├── atmos.yaml
├── components/terraform/mock/main.tf # empty file
└── stacks/orgs/
├── _defaults.yaml
├── a.yaml b.yaml c.yaml d.yaml e.yaml # filler stacks — each: import [./_defaults], vars.stage: , a mock component
└── target.yaml # the locals-bearing file
```

`atmos.yaml`:
```yaml
base_path: "./"
components:
terraform:
base_path: "components/terraform"
stacks:
base_path: "stacks"
included_paths: ["orgs/**/*"]
excluded_paths: ["**/_defaults.yaml"]
name_template: "{{ .vars.namespace }}-{{ .vars.stage }}"
templates:
settings:
enabled: true
```

`stacks/orgs/_defaults.yaml`:
```yaml
vars:
namespace: acme
```

`stacks/orgs/target.yaml`:
```yaml
import: [./_defaults]
vars:
namespace: acme # duplicated to work around #XXX (deriveStackName import-merge bug)
stage: target
locals:
# These fail without cloud auth; any state ref triggers eager eval.
vpc_id: !terraform.state vpc some-other-stack ".vpc_id"
components:
terraform:
app:
metadata: {component: mock}
vars: {vpc: "{{ .locals.vpc_id }}"}
```

Each of `a.yaml`..`e.yaml` (trivial stacks that don't use locals — just there to inflate the stack tree):
```yaml
import: [./_defaults]
vars:
stage: a # (or b, c, d, e)
components:
terraform:
app:
metadata: {component: mock}
```

Run:
```
$ ATMOS_LOGS_LEVEL=Debug atmos list stacks 2>&1 | grep -c 'function="!terraform.state'
```

Expected: `1` (or small constant).
Actual: scales with number of stacks × imports.

## Root cause (hypothesis)

`pkg/locals/resolver.go:resolveString` calls `yamlFunctionProcessor` eagerly for every resolution. During `list stacks`, Atmos visits every stack file; for each visit that touches the locals-bearing file (directly or transitively), the locals are re-resolved from scratch. No memoization.

The same `!terraform.state ` call with identical arguments always returns the same value — results are safe to cache for the lifetime of a single command invocation.

## Fix direction

Memoize resolved locals per `(filePath)` for the lifetime of a single Atmos command. Either in `pkg/locals/resolver.go` or at the caller site in `internal/exec/stack_processor_locals.go`.

## Affected versions

Reproduced on v1.216.0.

## Related

- Independent of #2343 (`deriveStackNameForLocals` import-merge bug) and #2345 (`describe-stacks-name-template` hardcoded `false`), but frequently hit together. In hierarchical layouts, users who work around #2343 (duplicate identity vars) still hit this performance issue.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/locals/resolver.go at resolveString and compare the caller path in internal/exec/stack_processor_locals.go. Reproduce with the supplied stack tree and `ATMOS_LOGS_LEVEL=Debug atmos list stacks 2>&1 | grep -c 'function="!terraform.state'`. Done means identical locals are resolved once per file for a command, with the count staying near a small constant instead of scaling with stacks and imports.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.