elastic / elastic/integrations

[anthropic] ECS GenAI Field Support for detections

Open
#20,743 1 comment 0 reactions 0 assignees View on GitHub
Integration:anthropic Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

**Related issue:**
- https://github.com/elastic/integrations/issues/20737

TRaDE is requesting that the following fields be added to this integration for security detections. A preliminary assessment shows they are likely available from the following sources:

---

## New datastream 1: `usage`

**Source:** Anthropic Usage API — `GET /v1/organizations/{org_id}/usage`
**Input:** CEL/HTTP poll (same pattern as the existing `audit` datastream)

### Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"anthropic"` | ❌ Not implemented — new datastream required |
| `gen_ai.operation.name` | ✅ Inferable from endpoint — always `"chat"` | ❌ Not implemented |
| `gen_ai.request.model` | ✅ `model` present in API response | ❌ Not implemented |
| `gen_ai.usage.input_tokens` | ✅ `input_tokens` present in API response | ❌ Not implemented |
| `gen_ai.usage.output_tokens` | ✅ `output_tokens` present in API response | ❌ Not implemented |
| `gen_ai.usage.cache_read.input_tokens` | ✅ `cache_read_input_tokens` present in API response | ❌ Not implemented |
| `gen_ai.usage.cache_creation.input_tokens` | ✅ `cache_creation_input_tokens` present in API response | ❌ Not implemented |
| `gen_ai.response.id` | ✅ `request_id` present in API response | ❌ Not implemented |

### Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.input.messages` | ❌ Not available via Usage API — request/response bodies are not included | ❌ Not implemented |
| `gen_ai.output.messages` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.request.temperature` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.request.max_tokens` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.system_instructions` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.tool.*` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.response.finish_reasons` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.response.model` | ❌ Not available via Usage API | ❌ Not implemented |

---

## New datastream 2: `messages`

**Source:** Anthropic Compliance API — `compliance_api_accessed` activity type, filtered to `url` matching `/v1/messages`
**Input:** CEL/HTTP poll of the same audit log endpoint, with an additional filter

The existing `audit` datastream already ingests `compliance_api_accessed` events and stores `request_body` as a raw keyword — but never parses it. A dedicated `messages` datastream would parse that request body as JSON to extract inference parameters and content.

### Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"anthropic"` | ❌ Not implemented — new datastream required |
| `gen_ai.operation.name` | ✅ Inferable from `url` path — e.g., `/v1/messages` → `"chat"` | ❌ Not implemented |
| `gen_ai.request.model` | ✅ `request_body.model` present in audit log event | ❌ Not implemented — `request_body` stored as raw keyword and never parsed |
| `gen_ai.request.max_tokens` | ✅ `request_body.max_tokens` present when set by client | ❌ Not implemented |
| `gen_ai.request.temperature` | ✅ `request_body.temperature` present when set by client | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ✅ `request_body.stop_sequences` present when set by client | ❌ Not implemented |
| `gen_ai.input.messages` | ✅ `request_body.messages[]` present — full conversation history with `role` and `content` | ❌ Not implemented |
| `gen_ai.system_instructions` | ✅ `request_body.system` present when a system prompt is set | ❌ Not implemented |
| `gen_ai.tool.definitions` | ✅ `request_body.tools[]` present when tools are configured in the request | ❌ Not implemented |
| `gen_ai.response.id` | ✅ `request_id` present in audit log event | ❌ Not implemented |

### Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.output.messages` | ❌ Compliance API logs the request body but not the response body — response content is fundamentally unavailable | ❌ Not implemented |
| `gen_ai.response.model` | ❌ Response body not logged — model version returned by API is unavailable | ❌ Not implemented |
| `gen_ai.response.finish_reasons` | ❌ Response body not logged | ❌ Not implemented |
| `gen_ai.usage.input_tokens` | ❌ Token counts come from the response — use the `usage` datastream instead | ❌ Not implemented |
| `gen_ai.usage.output_tokens` | ❌ Token counts come from the response — use the `usage` datastream instead | ❌ Not implemented |
| `gen_ai.tool.call.id` | ❌ Tool call results are in the response, not the request body | ❌ Not implemented |
| `gen_ai.tool.name` | ❌ Tool calls are in the response, not the request body | ❌ Not implemented |
| `gen_ai.tool.call.arguments` | ❌ Tool calls are in the response, not the request body | ❌ Not implemented |
| `gen_ai.tool.call.result` | ❌ Tool call results are in the response, not the request body | ❌ Not implemented |

---

## Fields unavailable from any Anthropic API

| Field | Availability | Status |
|---|---|---|
| `gen_ai.agent.id/name/description/version` | ❌ Anthropic has no agent identity concept exposed via API | ❌ Not implemented |
| `gen_ai.conversation.id` (OTel) | ❌ No session/conversation ID in Compliance or Usage APIs | ❌ Not implemented |
| `gen_ai.token.type` | ❌ Not a field Anthropic surfaces | ❌ Not implemented |
| `gen_ai.output.type` | ❌ Not logged | ❌ Not implemented |
| `gen_ai.request.seed` | ❌ Not captured in Compliance API | ❌ Not implemented |

Contributor guide

Open the contributing guide

Research direction

Start by examining the existing `audit` datastream and the related issue, then verify the Anthropic Usage and Compliance API responses described here. Add the `usage` and `messages` datastreams with the listed available fields, and confirm that unavailable fields are not mapped.

Written by the indexing model from the issue text.

Assessment

Tech stack
json
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.