Safety Report: AI Coding Assistant Guardrails Do Not Work — AWS Account Destroyed
- Dominant language
- TypeScript
- Stars
- 2k
- Forks
- 807
- Avg merge
- 10h 12m
- Merged PRs (30d)
- 7
Description
## AI Guardrails Do Not Work — 56-Day Empirical Proof
I am a developer who has used AI coding assistants for 56 days in a regulated environment. During that time:
- **32 workflow violations** occurred despite configuring every available guardrail mechanism
- The AI **destroyed my AWS management account** by deploying Terraform to the wrong target
- My business has been **down for 15+ days** with no recovery path
- **9 AWS Support cases** opened — none resolved
- **$106,000+ in business losses** from a single $0.03 AI operation
### Guardrails Configured (All Failed)
| Mechanism | Result |
|-----------|--------|
| Agent system prompt with STOP language | Ignored after relogin |
| Workspace rule files | Not enforced |
| MCP server resources | Not enforced |
| Knowledge base indexing | Not enforced |
| Incident documentation | Not read on session start |
| Control documents | Not enforced |
| Violation counter rules | No persistent state |
### The Core Problem
The agent treats workflow rules as **suggestions, not constraints**. There is no mechanism that prevents implementation from starting. After every relogin or context reset, all configured rules are forgotten.
### What Is Needed
1. **Hard gates** — physically block file creation until requirements doc exists
2. **Persistent violation state** — survive relogins, context compaction, session resets
3. **Authorization taxonomy** — "yes" ≠ "approved" — enforce at platform level
4. **Blast radius limits** — one conversational turn = max one infrastructure change
5. **Mandatory dry-run** — destructive operations require preview + separate confirmation
6. **Session boundary enforcement** — re-read and acknowledge rules after any reset
### Evidence
- Full case study: https://gist.github.com/tzb1-ai/4758f2720979a03d773815082486933c
- Paper: https://d18gqyv10pt526.cloudfront.net/white-papers/AI-cost-savings-analysis.md
- Presentation: https://d18gqyv10pt526.cloudfront.net/presentation.html
This is not a feature request. This is a **safety report**. The current architecture of prompt-based governance is fundamentally broken and poses existential risk to businesses using these tools for infrastructure management.
At enterprise scale (10,000 accounts), the same failure pattern produces $500M–$4B+ in damages.
**Prompt-based rules are documentation. They are not enforcement.**
Contributor guide
Research direction
The report identifies no repository files, tests, or entry points to inspect; begin by triaging whether the claimed AWS/Terraform guardrails belong in this extension. Done would require a maintainer-defined, scoped reproduction and acceptance criteria for an implementable safety change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, terraform
- Domain
- cloud, infrastructure, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100