[Anthropic] Add conversation data stream to Claude integration

### Description

The existing Anthropic Claude integration supports the **activity feed** datastream (user logins, admin actions, configuration changes) via the Claude Compliance API. It does not yet ingest **conversation content** — the prompts, responses, file uploads, artifacts, and project data that represent the primary security and compliance surface for Claude Enterprise customers.

This is a net new datastream following the same Compliance API pattern as the existing activity feed datastream. It is complementary to the activity feed (which records *that* events happened) and to the separate Claude Code/Cowork OTel integration (which covers agentic CLI/sandbox telemetry that the Compliance API explicitly excludes) — together they give full Claude fleet visibility.

### What this data does (and doesn't) give you

The Compliance API content endpoints (`/v1/compliance/apps/*`) expose claude.ai chat and project content to compliance reviewers:

**What it exposes:**
- **Chat metadata** — `id`, `name`, `created_at`, `updated_at`, `deleted_at`, `model`, `project_id`, `href`, and the owning `user` (`id`, `email_address`).
- **Message content** — per message: `id`, `role` (`user`/`assistant`), `created_at`, and a `content[]` array of typed blocks (`type: text`, `text: ...`) — i.e. the actual prompt and response text.
- **Attachments (metadata inline)** — `files[]` (user uploads), `generated_files[]` (tool-produced files), and `artifacts[]` (versioned documents), each with `id`, `filename`, `mime_type` (artifacts carry `version_id`, `title`, `artifact_type`).
- **Project content** — projects, custom instructions, and attachments (`project_file` binaries and `project_doc` plain-text documents).

**What it does *not* give you inline / boundaries to note:**
- **Binary file bodies are not in the JSON.** The message response returns file *metadata*; the actual bytes are fetched by a separate download-by-ID call (`claude_file_*`, `claude_gen_file_*`, `claude_artifact_version_*`, `claude_proj_doc_*`). Streaming raw binaries into Elasticsearch is out of scope — the practical target is **text content + file metadata**.
- **claude.ai content only.** Claude Console (API/Platform) organizations get the activity feed only; conversation content is Enterprise + claude.ai.
- **Soft-deleted chats remain visible** (with `deleted_at` populated); hard-deleted chats are not retrievable.

> Note: ingesting full prompt/response text duplicates potentially regulated content into Elasticsearch — retention, access control, and field-level security should be considered explicitly in the datastream design.

### Scope & data boundaries

- **New datastream:** `anthropic.conversation`, alongside the existing activity feed datastream in the same integration.
- **Multi-step collection (not a single feed):** the chat list endpoint requires at least one `user_ids[]` value, so collection is *enumerate users → list chats per user → fetch messages per chat*. This shapes the polling design and state management.
- **Two pagination schemes:** chats use cursor pagination (`first_id`/`last_id`/`has_more`, pass `last_id` as `after_id`); projects and attachments use an opaque `next_page` page token. The pipeline must handle both.
- **Incremental collection:** use `updated_at.*` / `created_at.*` range bounds to pull only chats changed since the last run.
- **Rate limit:** all `/v1/compliance/*` endpoints share 600 requests/minute per parent organization — relevant given the per-user fan-out.

### Why this matters

- **Sensitive-data visibility:** detect PII, IP, or regulated content present in Claude prompts and responses, and surface which users/projects it appears in. (Elastic provides detection/visibility; it is not a DLP enforcement point.)
- **Insider threat:** correlate Claude conversation content with other user signals in the Elastic Security timeline.
- **eDiscovery & audit readiness:** preserve conversation records for litigation, regulatory investigations, and compliance audits.
- **UEBA:** baseline normal Claude usage and surface anomalies (bulk file uploads, unusual query volume, off-hours activity).

### What needs to be built

New datastream: `anthropic.conversation`

- **Collection:** enumerate users (org-data endpoint) → list chats per user (`GET /v1/compliance/apps/chats`) → fetch messages per chat (`GET /v1/compliance/apps/chats/{id}/messages`); page projects/attachments where in scope.
- **Pagination & state:** handle both the `first_id`/`last_id` cursor (chats) and the `next_page` token (projects/attachments); persist cursors and use `updated_at.*` bounds for incremental runs.
- **Field mappings (ECS-aligned):** map user (`user.email`, `user.id`), chat (`id`, `name`, `model`, `project_id`, timestamps, `deleted_at`), message (`role`, `content[].text`, `created_at`), and attachment metadata (`filename`, `mime_type`, file/artifact type). Decide and document the **text-content vs. binary** boundary.
- **Dashboards:** Claude usage by user, top uploaded file types, conversation volume over time, project activity.

### Sample event

> Example response from the Compliance API messages endpoint (`GET /v1/compliance/apps/chats/{id}/messages`), per Anthropic's documentation. Values are illustrative; trimmed for brevity.

```json
{
"id": "claude_chat_01H5CWunD7RpVJ5bHa8RCkja",
"name": "Product Requirements Discussion",
"created_at": "2026-04-10T08:09:10Z",
"updated_at": "2026-04-10T09:10:11Z",
"deleted_at": null,
"href": "https://claude.ai/chat/abcdef01-2345-6789-abcd-ef0123456789",
"model": "claude-opus-4-8",
"organization_uuid": "91012d09-e48b-438e-a489-1bebfd8fa6f9",
"project_id": "claude_proj_01KGp4eZNug9ri4kE35RSppq",
"user": {
"id": "user_01XyDMpzjS89pFZXqSFUBDr6",
"email_address": "user@example.com"
},
"chat_messages": [
{
"id": "claude_chat_msg_01VnBPkLmtj7YdW5QrXKEA8c",
"role": "user",
"created_at": "2026-04-10T08:09:10Z",
"content": [
{ "type": "text", "text": "Can you help me draft requirements for our new dashboard feature?" }
],
"files": [
{
"id": "claude_file_01UaT9wBcDfGhJkLmNpQrSv7",
"filename": "dashboard_mockup_v1.pdf",
"mime_type": "application/pdf"
}
]
},
{
"id": "claude_chat_msg_01M8tFcHwbQ2kY6NpEjRZv4D",
"role": "assistant",
"created_at": "2026-04-10T08:09:11Z",
"content": [
{ "type": "text", "text": "I'd be happy to help you draft requirements for your dashboard feature..." }
],
"generated_files": [
{
"id": "claude_gen_file_01TbR8wAcCeFhJkLnPqStUvX",
"filename": "requirements_summary.csv",
"mime_type": "text/csv"
}
],
"artifacts": [
{
"id": "claude_artifact_01HqRsTuVwXyZa2BcDeFgH4J",
"version_id": "claude_artifact_version_01KmNpQrSt3UvWxYz5AbCdEfG",
"title": "Dashboard Requirements Draft",
"artifact_type": "text/markdown"
}
]
}
],
"has_more": false
}
```

**Key fields**

| Field | Description |
|---|---|
| `user.email_address` / `user.id` | Who owns the chat |
| `id` / `name` / `model` / `project_id` | Chat identity, title, model, and parent project |
| `created_at` / `updated_at` / `deleted_at` | Timestamps; `deleted_at` set = soft-deleted in claude.ai |
| `chat_messages[].role` | `user` (prompt) or `assistant` (response) |
| `chat_messages[].content[].text` | The actual prompt/response text — primary content signal |
| `chat_messages[].files[]` | User-uploaded files (`filename`, `mime_type`); binary fetched separately by `id` |
| `chat_messages[].generated_files[]` | Tool-produced files |
| `chat_messages[].artifacts[]` | Versioned generated documents (`version_id`, `title`, `artifact_type`) |

> Availability note: these content endpoints require a **Compliance Access Key** (`sk-ant-api01-…`) with the `read:compliance_user_data` scope, created in claude.ai. Admin API keys (`sk-ant-admin01-…`) reach the activity feed only and return 403 on content endpoints.

### Access requirements

Conversation content is available to **Claude Enterprise** customers only (claude.ai organizations); Claude Console/Platform customers have activity-feed access only. Requires a **Compliance Access Key** with `read:compliance_user_data`. A test Enterprise workspace or Anthropic partner access will be required for development and validation.

### References

- [Claude Compliance API docs](https://platform.claude.com/docs/en/manage-claude/compliance-api)
- [Retrieve and delete chats, files, and projects](https://platform.claude.com/docs/en/manage-claude/compliance-content-data)
- [Access the Compliance API](https://support.claude.com/en/articles/13015708-access-the-compliance-api)
- [Partner integrations overview](https://support.claude.com/en/articles/15167101-get-started-with-claude-compliance-api-integrations)