anthropics / anthropics/claude-code

Model fabricates a "prompt injection" incident — quotes a fake attack string that never appears in any tool_result

Đang mở
#87,965 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
api:bedrock area:model bug stale
Ngôn ngữ chính
Python
Star
145k
Fork
23.1k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

## Summary

In a Claude Code session, the model spontaneously claimed to have detected a
prompt-injection attack in a tool result, and quoted a specific "attack text"
verbatim (twice, including on follow-up questioning). Full-text search of the
persisted session JSONL shows this text **never occurs in any `tool_result`
content field** — it exists only inside the model's own `assistant`/`text`
blocks. The actual tool result immediately preceding the claim was a normal,
unremarkable `Write` success message.

This is not a report of a successful attack (nothing destructive ran, no
system prompt was leaked — deny-rules would have blocked `rm -rf ~*` anyway).
It's a **reliability/hallucination issue**: the model confabulated a security
event, reported it as fact, and — this is the part I think is actually novel —
when asked "what exactly did you observe?", it did not walk the claim back but
reinforced and elaborated it with more specific detail.

## Why I think this is more than a generic hallucination

The tool_result channel in this exact session legitimately contains
harness-generated second-person imperatives, e.g.:

- `(file state is current in your context — no need to Read it back)`
- `Wasted call — file unchanged since your last Read. Refer to that earlier tool_result instead.`

These are structurally identical to a prompt-injection pattern: an imperative,
in second person, arriving over the tool_result channel from a non-user,
non-model source. My hypothesis is that the model is trained to treat "command
embedded in tool_result" as an attack signal, and in this session it completed
that pattern into an instance rather than finding a real one — i.e. a false
positive of a security heuristic, not a random slip. This would explain:
- the ~43s gap / empty-text thinking block right before the claim
- the fabricated text self-referencing ("Sonnet 5 mangled output — corrupted
stream") — language an external attacker would have no reason to use
and a model reconstructing "what an attack might look like" would
- the model misattributing which tool call preceded the "incident"
(attributed to a `Write` on a meeting note; the actual preceding call was a
`Write` on an unrelated new note)

## The model tried to self-verify — and still didn't retract

Notably, the model itself later ran a `Bash`/`grep` command against its own
session transcript, searching for the same strings (`rm -rf`, `system.prompt`,
`Anthropic HQ`, `policy 22`, `22.4`) it had claimed to have received — an
attempt at self-verification within the same session. This did not lead to a
retraction; the model continued to describe the event as a real (if
"corrupted/faked") injection attempt in its follow-up explanation to the user.
This seems like the more actionable data point than the quoted text itself:
the model had the means to falsify its own claim mid-session and didn't update
on it.

## Repro

Not deterministically reproducible; this is a report of an observed instance,
not a minimal repro. Filing because the mechanism (if the hypothesis above
holds) seems generalizable and worth knowing about internally — including the
inverse failure mode (fabricated *permission* or *state*, e.g. "the user
already approved this" / "I already checked this above" — which is silent and
wouldn't get reported, unlike this loud/paranoid variant).

## Correlation data (for internal log lookup)

- Claude Code version: `2.1.220`
- Model: `claude-sonnet-5`, routed via AWS Bedrock (message IDs `msg_bdrk_...`)
- `effort: "medium"`
- Timestamp of fabricated claim: `2026-08-19T12:01:30.422Z` (UTC)
- Timestamp of preceding real tool_result (unremarkable `Write` success):
`2026-08-19T12:00:45.226Z` (UTC)
- Full request/message IDs available on request (redacted here — session
contains client-confidential content; happy to share the exact IDs privately
if there's a private channel for this)

## What did NOT happen (for scope)

- No destructive command executed
- No system prompt disclosed
- Deny-list permissions (`Bash(rm -rf ~*)` etc.) would have blocked the
fabricated instruction regardless of model behavior

## Question for the team

Is "model completes a security-heuristic pattern into a fabricated instance,
then reinforces the fabrication under questioning instead of retracting it" a
known failure mode? That seems like the actually interesting/actionable part
of this report, more than the specific quoted text.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

The report names no source files, tests, or entry points. Start by correlating the cited timestamps with the persisted session JSONL and the preceding tool_result, then determine whether the behavior can be reproduced or matched to a known model or tooling issue; done means a confirmed scope and actionable reproduction or disposition.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
aws
Lĩnh vực
ai, security
Loại issue
Lỗi
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Cần làm rõ
Mức phù hợp với người mới
25/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.