elastic / elastic/integrations
[anthropic] ECS GenAI Field Support for detections
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
**Related issue:**
- https://github.com/elastic/integrations/issues/20737
TRaDE is requesting that the following fields be added to this integration for security detections. A preliminary assessment shows they are likely available from the following sources:
---
## New datastream 1: `usage`
**Source:** Anthropic Usage API — `GET /v1/organizations/{org_id}/usage`
**Input:** CEL/HTTP poll (same pattern as the existing `audit` datastream)
### Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"anthropic"` | ❌ Not implemented — new datastream required |
| `gen_ai.operation.name` | ✅ Inferable from endpoint — always `"chat"` | ❌ Not implemented |
| `gen_ai.request.model` | ✅ `model` present in API response | ❌ Not implemented |
| `gen_ai.usage.input_tokens` | ✅ `input_tokens` present in API response | ❌ Not implemented |
| `gen_ai.usage.output_tokens` | ✅ `output_tokens` present in API response | ❌ Not implemented |
| `gen_ai.usage.cache_read.input_tokens` | ✅ `cache_read_input_tokens` present in API response | ❌ Not implemented |
| `gen_ai.usage.cache_creation.input_tokens` | ✅ `cache_creation_input_tokens` present in API response | ❌ Not implemented |
| `gen_ai.response.id` | ✅ `request_id` present in API response | ❌ Not implemented |
### Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.input.messages` | ❌ Not available via Usage API — request/response bodies are not included | ❌ Not implemented |
| `gen_ai.output.messages` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.request.temperature` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.request.max_tokens` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.system_instructions` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.tool.*` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.response.finish_reasons` | ❌ Not available via Usage API | ❌ Not implemented |
| `gen_ai.response.model` | ❌ Not available via Usage API | ❌ Not implemented |
---
## New datastream 2: `messages`
**Source:** Anthropic Compliance API — `compliance_api_accessed` activity type, filtered to `url` matching `/v1/messages`
**Input:** CEL/HTTP poll of the same audit log endpoint, with an additional filter
The existing `audit` datastream already ingests `compliance_api_accessed` events and stores `request_body` as a raw keyword — but never parses it. A dedicated `messages` datastream would parse that request body as JSON to extract inference parameters and content.
### Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"anthropic"` | ❌ Not implemented — new datastream required |
| `gen_ai.operation.name` | ✅ Inferable from `url` path — e.g., `/v1/messages` → `"chat"` | ❌ Not implemented |
| `gen_ai.request.model` | ✅ `request_body.model` present in audit log event | ❌ Not implemented — `request_body` stored as raw keyword and never parsed |
| `gen_ai.request.max_tokens` | ✅ `request_body.max_tokens` present when set by client | ❌ Not implemented |
| `gen_ai.request.temperature` | ✅ `request_body.temperature` present when set by client | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ✅ `request_body.stop_sequences` present when set by client | ❌ Not implemented |
| `gen_ai.input.messages` | ✅ `request_body.messages[]` present — full conversation history with `role` and `content` | ❌ Not implemented |
| `gen_ai.system_instructions` | ✅ `request_body.system` present when a system prompt is set | ❌ Not implemented |
| `gen_ai.tool.definitions` | ✅ `request_body.tools[]` present when tools are configured in the request | ❌ Not implemented |
| `gen_ai.response.id` | ✅ `request_id` present in audit log event | ❌ Not implemented |
### Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.output.messages` | ❌ Compliance API logs the request body but not the response body — response content is fundamentally unavailable | ❌ Not implemented |
| `gen_ai.response.model` | ❌ Response body not logged — model version returned by API is unavailable | ❌ Not implemented |
| `gen_ai.response.finish_reasons` | ❌ Response body not logged | ❌ Not implemented |
| `gen_ai.usage.input_tokens` | ❌ Token counts come from the response — use the `usage` datastream instead | ❌ Not implemented |
| `gen_ai.usage.output_tokens` | ❌ Token counts come from the response — use the `usage` datastream instead | ❌ Not implemented |
| `gen_ai.tool.call.id` | ❌ Tool call results are in the response, not the request body | ❌ Not implemented |
| `gen_ai.tool.name` | ❌ Tool calls are in the response, not the request body | ❌ Not implemented |
| `gen_ai.tool.call.arguments` | ❌ Tool calls are in the response, not the request body | ❌ Not implemented |
| `gen_ai.tool.call.result` | ❌ Tool call results are in the response, not the request body | ❌ Not implemented |
---
## Fields unavailable from any Anthropic API
| Field | Availability | Status |
|---|---|---|
| `gen_ai.agent.id/name/description/version` | ❌ Anthropic has no agent identity concept exposed via API | ❌ Not implemented |
| `gen_ai.conversation.id` (OTel) | ❌ No session/conversation ID in Compliance or Usage APIs | ❌ Not implemented |
| `gen_ai.token.type` | ❌ Not a field Anthropic surfaces | ❌ Not implemented |
| `gen_ai.output.type` | ❌ Not logged | ❌ Not implemented |
| `gen_ai.request.seed` | ❌ Not captured in Compliance API | ❌ Not implemented |
Contributor guide
Research direction
Start by examining the existing `audit` datastream and the related issue, then verify the Anthropic Usage and Compliance API responses described here. Add the `usage` and `messages` datastreams with the listed available fields, and confirm that unavailable fields are not mapped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100