aws / aws/aws-toolkit-vscode

Safety Report: AI Coding Assistant Guardrails Do Not Work — AWS Account Destroyed

Open
#8,799 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2k
Forks
807
Avg merge
10h 12m
Merged PRs (30d)
7

Description

## AI Guardrails Do Not Work — 56-Day Empirical Proof

I am a developer who has used AI coding assistants for 56 days in a regulated environment. During that time:

- **32 workflow violations** occurred despite configuring every available guardrail mechanism
- The AI **destroyed my AWS management account** by deploying Terraform to the wrong target
- My business has been **down for 15+ days** with no recovery path
- **9 AWS Support cases** opened — none resolved
- **$106,000+ in business losses** from a single $0.03 AI operation

### Guardrails Configured (All Failed)

| Mechanism | Result |
|-----------|--------|
| Agent system prompt with STOP language | Ignored after relogin |
| Workspace rule files | Not enforced |
| MCP server resources | Not enforced |
| Knowledge base indexing | Not enforced |
| Incident documentation | Not read on session start |
| Control documents | Not enforced |
| Violation counter rules | No persistent state |

### The Core Problem

The agent treats workflow rules as **suggestions, not constraints**. There is no mechanism that prevents implementation from starting. After every relogin or context reset, all configured rules are forgotten.

### What Is Needed

1. **Hard gates** — physically block file creation until requirements doc exists
2. **Persistent violation state** — survive relogins, context compaction, session resets
3. **Authorization taxonomy** — "yes" ≠ "approved" — enforce at platform level
4. **Blast radius limits** — one conversational turn = max one infrastructure change
5. **Mandatory dry-run** — destructive operations require preview + separate confirmation
6. **Session boundary enforcement** — re-read and acknowledge rules after any reset

### Evidence

- Full case study: https://gist.github.com/tzb1-ai/4758f2720979a03d773815082486933c
- Paper: https://d18gqyv10pt526.cloudfront.net/white-papers/AI-cost-savings-analysis.md
- Presentation: https://d18gqyv10pt526.cloudfront.net/presentation.html

This is not a feature request. This is a **safety report**. The current architecture of prompt-based governance is fundamentally broken and poses existential risk to businesses using these tools for infrastructure management.

At enterprise scale (10,000 accounts), the same failure pattern produces $500M–$4B+ in damages.

**Prompt-based rules are documentation. They are not enforcement.**

Contributor guide

Open the contributing guide

Research direction

The report identifies no repository files, tests, or entry points to inspect; begin by triaging whether the claimed AWS/Terraform guardrails belong in this extension. Done would require a maintainer-defined, scoped reproduction and acceptance criteria for an implementable safety change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, terraform
Domain
cloud, infrastructure, security
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.