cloudposse / cloudposse/atmos

atmos describe affected: nested YAML functions in cross-stack !terraform.state target resolve in caller's stack context instead of target's

Open
#2,360 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.4k
Forks
175
Avg merge
2d 3h
Merged PRs (30d)
134

Description

## Describe the Bug

When `atmos describe affected` evaluates a cross-stack `!terraform.state ` reference (3-arg form), atmos pulls the target component's merged YAML config and evaluates any nested YAML functions it contains. Those nested functions appear to resolve against the **caller's** stack context rather than the **target's** stack context, producing false `Could not find the component X in the stack Y` errors whenever a same-stack lookup in the target (e.g. `!terraform.state kms key_arn`) refers to a component that exists in the target's stack but not in the caller's.

This surfaces only when `ci.enabled: true` and the new zero-config CI base detection from #2241 are both in effect — i.e. when `describe affected --upload` is running without explicit `--ref` / `--sha` / `--base` and with function evaluation enabled (the default, equivalent to `--process-functions=true`).

## Expected Behavior

When atmos evaluates a nested `!terraform.state ` (two-arg, same-stack) call that appears inside a component whose outputs are being resolved cross-stack, the nested call should resolve against the component's **own** stack — not the caller's. In practice, that means:

- Caller stack: `stack-a` references `!terraform.state target-component stack-b output`.
- Target component in `stack-b` has `kms_key_id: !terraform.state kms key_arn` in its merged config.
- Resolving the outer 3-arg reference requires reading state from `stack-b`. If atmos walks the target's config to do that, any nested same-stack lookups should resolve against `stack-b` (where `kms` exists), not against `stack-a`.

## Current Behavior

Nested same-stack `!terraform.state ` calls inside the target's merged config are evaluated against the caller's stack context. When the referenced component doesn't exist in the caller's stack, `describe affected` fails with:

```
failed to describe component kms in stack in YAML function:
!terraform.state kms key_arn invalid component: Could not find the component kms in
the stack . Check that all the context variables are correctly defined
in the stack manifests. Are the component and stack names correct? Did you forget an
import?
```

The `caller-stack` in the error is the stack that initiated the describe-affected walk (e.g. a PR branch's affected-components output), not the stack the nested reference lives in.

## Minimal Reproduction

Two stacks and three components:

1. Component `kms` deployed **only** in stack `stack-b`.
2. Component `secrets-manager/shared` deployed in `stack-b`. Its merged config contains a same-stack reference: `kms_key_id: !terraform.state kms key_arn`.
3. Component `lambda/consumer` deployed in `stack-a`. Its config references the secrets-manager component cross-stack:
```yaml
CREDS_ARN: !terraform.state secrets-manager/shared stack-b secret_arns_map.shared/credentials
```

Run with `ci.enabled: true`:

```bash
atmos describe affected --upload
# or equivalently
atmos describe affected --upload --process-functions=true
```

On a PR that modifies the stack-wide config (e.g. an `_defaults.yaml` edit that makes `stack-a` appear as affected), atmos fails with:

```
failed to describe component kms in stack stack-a in YAML function:
!terraform.state kms key_arn
```

Even though `kms` is not referenced by any component in `stack-a` directly. The reference being evaluated is the nested one inside `secrets-manager/shared` (which lives in `stack-b` and does have `kms` co-located).

## Workaround

Pass `--process-functions=false` to `atmos describe affected`:

```bash
atmos describe affected --upload --process-functions=false
```

This skips all YAML function evaluation, which is acceptable for describe-affected (the command only needs stack-level change detection, not resolved values) but is a deviation from the canonical CI pattern shown at https://github.com/cloudposse-examples/atmos-pro-example-advanced where the flag is omitted.

## Environment

- atmos `v1.216.0` (also reproduces with `v1.215.0`)
- `ci.enabled: true` in the active profile
- `provider=github-actions`, `event=pull_request` (zero-config CI base detection, #2241)
- Workflow does `atmos describe affected --upload` against a PR where the affected-stacks set includes a stack whose components transitively reference a cross-stack component whose merged config contains a same-stack `!terraform.state` to a component not deployed in the caller's stack.

## Why This Matters

Any repo that uses the canonical atmos-pro workflow pattern *and* has cross-stack `!terraform.state` references where the target's config contains same-stack nested functions will hit this. The failure mode isn't immediately obvious from the error text (the error names a "missing" component that doesn't appear in the caller's stack config at all — it's only present in the target's).

The same pattern works fine at `terraform apply` time because terraform itself resolves `!terraform.state` cleanly via backend state reads without re-evaluating the target's YAML functions. The bug is specific to the describe-affected YAML evaluation path.

## Suggested Fix

When resolving a 3-arg cross-stack `!terraform.state`, scope any nested YAML function evaluation to the **target** stack's context (the stack named by the 2nd arg), not the stack that initiated the describe-affected walk.

Contributor guide

Open the contributing guide

Research direction

Start with `atmos describe affected --upload` using the minimal two-stack reproduction and `--process-functions=true`. Trace the `!terraform.state` 3-argument evaluation path and its nested function handling. Done means nested same-stack lookups use the target stack context, while the described CI scenario succeeds without requiring `--process-functions=false`.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, go, terraform
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.