EKS auto-provisioning: redundant DescribeCluster calls across atmos subprocesses (workflow pattern)
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 175
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 134
Description
## Context
This issue captures an architectural observation discovered while investigating EKS auto-provisioning noise (see #2402 for the original noise fix). It's not blocking anything — #2402 already solved the user-visible problem of duplicate output. This issue tracks an underlying inefficiency that an in-process cache cannot solve.
## Observation
When atmos identities with linked \`aws/eks\` integrations are resolved across components in a workflow, each component triggers its own \`EKS:DescribeCluster\` API round trip even though the same cluster metadata has been fetched moments earlier in a sibling subprocess.
This happens because the dominant atmos workflow pattern shells out per component:
\`\`\`yaml
workflows:
plan/dev:
steps:
- command: |
atmos list components -s \"\$STACK\" \\
| awk '\$1 ~ /^bedrock\\/iam-user\\// {print \$1}' \\
| while read -r inst; do
atmos terraform plan \"\$inst\" -s \"\$STACK\"
done
\`\`\`
Each \`atmos terraform plan -s \` is a fresh OS process. Any in-memory cache (e.g., \`sync.Map\` keyed on cluster metadata) is destroyed at process exit. Real-world empirical data from a typical Atmos auth setup: 16 \`EKSIntegration.Execute\` calls produced **6 unique** \`(account, cluster, region)\` tuples but **zero** in-process cache hits, because each subprocess sees only its own first call.
## What would help
A **disk-backed**, per-(account, cluster, region) cache of \`DescribeCluster\` responses, with:
- **Concurrency-safe writes** (file locks; multiple atmos subprocesses can be running in parallel under \`-parallel\` workflows)
- **TTL** (cluster metadata can change — CA rotation, endpoint changes after re-creation; suggested default ~24h)
- **Cache key:** \`(AWS account ID, cluster name, region)\` — same boundary EKS uses for cluster uniqueness; account ID resolvable via \`sts:GetCallerIdentity\` on the credentials, with that result itself memoized
- **Invalidation hook** on \`atmos auth logout\` and probably on \`atmos aws eks update-kubeconfig --integration \` (so users have a way to force-refresh)
- **Storage location:** something XDG-compliant under \`~/.cache/atmos/eks-describe/\` to keep it separate from kubeconfig itself
## What's not enough
#2403 was an attempt to address this with a process-local \`sync.Map\` cache keyed on \`(account, cluster, region)\`, with \`DescribeCluster\` memoization across all identities in the same account. The empirical test against the typical \`atmos workflow ... atmos terraform plan ...\` shell-out pattern produced **zero cache hits** because each subprocess is its own process. The PR was closed; see #2403 for full design history including four adversarial-review rounds with codex that ultimately determined the in-process cache cannot help workflow subprocess patterns.
## Scope of work (rough)
- New package \`pkg/auth/cache/\` (or similar) for shared on-disk cache backend
- File-locking abstraction (look at \`gofrs/flock\` or \`pkg/store\` patterns)
- Migrate \`pkg/auth/integrations/aws/eks.go::describeClusterCached\` to use it
- Probably useful for other identity-related lookups too (e.g., \`sts:GetCallerIdentity\` results, \`aws sso\` token caching beyond what's already cached, ECR registry lookups)
- Migration story: cold-cache behavior is identical to current behavior, so this is purely additive
## Trade-offs to think through
- Disk cache + concurrent subprocesses = race conditions. Need locking strategy or write-the-same-data-is-okay semantics (\`DescribeCluster\` is deterministic so two parallel writes producing identical files is fine; the failure mode is partial writes during interrupt).
- Staleness window vs. correctness: a cached endpoint can outlive the cluster (rare). A force-refresh flag and short-ish default TTL (12-24h) probably right.
- Security: cluster metadata is not secret (endpoint, ARN, CA cert) but a disk cache shouldn't be world-readable.
- Test surface: meaningfully bigger than in-process — concurrency tests, TTL tests, lock-contention tests, corruption-recovery tests.
## Why this is not P0
The original user complaint was 1000+ noisy \`✓ EKS kubeconfig:\` lines in a single workflow. PR #2402 cut that to ~13 lines via no-op detection. The remaining inefficiency is invisible (AWS API calls happen quietly, behind the scenes) and small in absolute terms (~50-200ms per \`DescribeCluster\` call × N components per workflow). It's a real optimization to make eventually, but the user-facing pain was already addressed.
## References
- #2402 — EKS kubeconfig no-op noise suppression (the user-facing fix, already merged-or-mergeable)
- #2403 — In-process cache attempt that revealed this architectural limitation (closed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Contributor guide
Research direction
Start with pkg/auth/integrations/aws/eks.go::describeClusterCached and review pkg/store patterns plus #2403's design history. Trace how atmos terraform plan subprocesses invoke the integration; done means a concurrency-safe, TTL-bound disk cache with logout/update-kubeconfig invalidation and tests for concurrency, TTL, locking, and corruption recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go, kubernetes
- Domain
- cli, cloud, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100