cloudposse / cloudposse/atmos

EKS auto-provisioning: redundant DescribeCluster calls across atmos subprocesses (workflow pattern)

Open
#2,404 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.4k
Forks
175
Avg merge
2d 3h
Merged PRs (30d)
134

Description

## Context

This issue captures an architectural observation discovered while investigating EKS auto-provisioning noise (see #2402 for the original noise fix). It's not blocking anything — #2402 already solved the user-visible problem of duplicate output. This issue tracks an underlying inefficiency that an in-process cache cannot solve.

## Observation

When atmos identities with linked \`aws/eks\` integrations are resolved across components in a workflow, each component triggers its own \`EKS:DescribeCluster\` API round trip even though the same cluster metadata has been fetched moments earlier in a sibling subprocess.

This happens because the dominant atmos workflow pattern shells out per component:

\`\`\`yaml
workflows:
plan/dev:
steps:
- command: |
atmos list components -s \"\$STACK\" \\
| awk '\$1 ~ /^bedrock\\/iam-user\\// {print \$1}' \\
| while read -r inst; do
atmos terraform plan \"\$inst\" -s \"\$STACK\"
done
\`\`\`

Each \`atmos terraform plan -s \` is a fresh OS process. Any in-memory cache (e.g., \`sync.Map\` keyed on cluster metadata) is destroyed at process exit. Real-world empirical data from a typical Atmos auth setup: 16 \`EKSIntegration.Execute\` calls produced **6 unique** \`(account, cluster, region)\` tuples but **zero** in-process cache hits, because each subprocess sees only its own first call.

## What would help

A **disk-backed**, per-(account, cluster, region) cache of \`DescribeCluster\` responses, with:

- **Concurrency-safe writes** (file locks; multiple atmos subprocesses can be running in parallel under \`-parallel\` workflows)
- **TTL** (cluster metadata can change — CA rotation, endpoint changes after re-creation; suggested default ~24h)
- **Cache key:** \`(AWS account ID, cluster name, region)\` — same boundary EKS uses for cluster uniqueness; account ID resolvable via \`sts:GetCallerIdentity\` on the credentials, with that result itself memoized
- **Invalidation hook** on \`atmos auth logout\` and probably on \`atmos aws eks update-kubeconfig --integration \` (so users have a way to force-refresh)
- **Storage location:** something XDG-compliant under \`~/.cache/atmos/eks-describe/\` to keep it separate from kubeconfig itself

## What's not enough

#2403 was an attempt to address this with a process-local \`sync.Map\` cache keyed on \`(account, cluster, region)\`, with \`DescribeCluster\` memoization across all identities in the same account. The empirical test against the typical \`atmos workflow ... atmos terraform plan ...\` shell-out pattern produced **zero cache hits** because each subprocess is its own process. The PR was closed; see #2403 for full design history including four adversarial-review rounds with codex that ultimately determined the in-process cache cannot help workflow subprocess patterns.

## Scope of work (rough)

- New package \`pkg/auth/cache/\` (or similar) for shared on-disk cache backend
- File-locking abstraction (look at \`gofrs/flock\` or \`pkg/store\` patterns)
- Migrate \`pkg/auth/integrations/aws/eks.go::describeClusterCached\` to use it
- Probably useful for other identity-related lookups too (e.g., \`sts:GetCallerIdentity\` results, \`aws sso\` token caching beyond what's already cached, ECR registry lookups)
- Migration story: cold-cache behavior is identical to current behavior, so this is purely additive

## Trade-offs to think through

- Disk cache + concurrent subprocesses = race conditions. Need locking strategy or write-the-same-data-is-okay semantics (\`DescribeCluster\` is deterministic so two parallel writes producing identical files is fine; the failure mode is partial writes during interrupt).
- Staleness window vs. correctness: a cached endpoint can outlive the cluster (rare). A force-refresh flag and short-ish default TTL (12-24h) probably right.
- Security: cluster metadata is not secret (endpoint, ARN, CA cert) but a disk cache shouldn't be world-readable.
- Test surface: meaningfully bigger than in-process — concurrency tests, TTL tests, lock-contention tests, corruption-recovery tests.

## Why this is not P0

The original user complaint was 1000+ noisy \`✓ EKS kubeconfig:\` lines in a single workflow. PR #2402 cut that to ~13 lines via no-op detection. The remaining inefficiency is invisible (AWS API calls happen quietly, behind the scenes) and small in absolute terms (~50-200ms per \`DescribeCluster\` call × N components per workflow). It's a real optimization to make eventually, but the user-facing pain was already addressed.

## References

- #2402 — EKS kubeconfig no-op noise suppression (the user-facing fix, already merged-or-mergeable)
- #2403 — In-process cache attempt that revealed this architectural limitation (closed)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with pkg/auth/integrations/aws/eks.go::describeClusterCached and review pkg/store patterns plus #2403's design history. Trace how atmos terraform plan subprocesses invoke the integration; done means a concurrency-safe, TTL-bound disk cache with logout/update-kubeconfig invalidation and tests for concurrency, TTL, locking, and corruption recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go, kubernetes
Domain
cli, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.