elastic / elastic/integrations

[openai_chatgpt_enterprise] ECS GenAI Field Support for detections

Open
#20,898 1 comment 0 reactions 0 assignees View on GitHub
9.6 candidate enhancement Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

**Related issues:**
- https://github.com/elastic/integrations/issues/20737
- https://github.com/elastic/integrations/issues/19399

TRaDE is requesting comprehensive ECS GenAI field support across the `openai_chatgpt_enterprise` integration. Multiple data stream PRs are already in flight (#20205, #20208, #20209, #20201) but the gen_ai field coverage in each is incomplete.

---

## `app_log` data stream (PR #20205)

Captures connector/tool invocations — request and response pairs between ChatGPT Enterprise and MCP or service connectors.

| Field | Source in raw data | Status |
|---|---|---|
| `gen_ai.provider.name` | Hardcode `"openai"` | ✅ Implemented |
| `gen_ai.conversation.id` | `conversation_id` | ❌ Not mapped |
| `gen_ai.operation.name` | Hardcode `"invoke_tool"` | ❌ Not mapped |
| `gen_ai.tool.name` | `app_name` (e.g. `"Slack"`, `"gdrive"`) | ❌ Not mapped |
| `gen_ai.tool.type` | `app_type`: `"MCP"` → `"extension"`, `"SERVICE"` → `"function"` | ❌ Not mapped |
| `gen_ai.tool.call.arguments` | `input` object (request events) | ❌ Not mapped |
| `gen_ai.tool.call.result` | `output[].response.result` (response events) | ❌ Not mapped |

---

## `conversation_message` data stream (PR #20208)

Captures individual user and assistant messages in ChatGPT Enterprise conversations. Message text is present in `message.content.value`.

| Field | Source in raw data | Status |
|---|---|---|
| `gen_ai.provider.name` | Hardcode `"openai"` | ⚠️ **Bug** — PR sets `gen_ai.system = "openai"` which is not a valid ECS field |
| `gen_ai.response.model` | `message.author.model` (assistant messages) | ✅ Implemented |
| `gen_ai.conversation.id` | `conversation.id` | ❌ Not mapped |
| `gen_ai.operation.name` | Hardcode `"chat"` | ❌ Not mapped |
| `gen_ai.input.messages` | `message.content.value` where `message.author.type == "user"` | ❌ Not mapped |
| `gen_ai.output.messages` | `message.content.value` where `message.author.type == "assistant"` | ❌ Not mapped |
| `gen_ai.response.id` | `message.id` (assistant messages) | ❌ Not mapped |

---

## `custom_agents_log` data stream (PR #20209)

Captures agent lifecycle events (AGENT_CREATED, AGENT_RUN_CREATED) and connector call events (CONNECTOR_CALL_REQUESTED, CONNECTOR_CALL_COMPLETED).

| Field | Source in raw data | Status |
|---|---|---|
| `gen_ai.agent.name` | `event_details.agent_fields.name` | ✅ Implemented |
| `gen_ai.agent.id` | `event_details.agent_id` | ✅ Implemented |
| `gen_ai.tool.name` | `event_details.connector_name` (CONNECTOR_CALL events) | ✅ Implemented |
| `gen_ai.request.model` | `event_details.agent_fields.model_spec.name` | ✅ Implemented |
| `gen_ai.provider.name` | Hardcode `"openai"` | ❌ Not mapped |
| `gen_ai.operation.name` | Derivable from `event_type`: `AGENT_RUN_CREATED` → `"chat"`, `CONNECTOR_CALL_*` → `"invoke_tool"` | ❌ Not mapped |
| `gen_ai.tool.call.id` | `event_details.tool_call_id` (CONNECTOR_CALL events) | ❌ Not mapped |
| `gen_ai.tool.type` | Inferable as `"function"` for connector calls | ❌ Not mapped |
| `gen_ai.conversation.id` | `event_details.conversation_id` | ❌ Not mapped |

---

## `codex_log` data stream (PR #20201)

Captures Codex task events including model usage and tool calls.

| Field | Source in raw data | Status |
|---|---|---|
| `gen_ai.request.model` | `event_details.model` | ✅ Implemented |
| `gen_ai.usage.input_tokens` | `event_details.token_usage.input_tokens` | ✅ Implemented |
| `gen_ai.usage.output_tokens` | `event_details.token_usage.output_tokens` | ✅ Implemented |
| `gen_ai.tool.name` | `event_details.tool` (tool_call events) | ✅ Implemented |
| `gen_ai.provider.name` | Hardcode `"openai"` | ❌ Not mapped |
| `gen_ai.operation.name` | Derivable from `event_details.action` | ❌ Not mapped |
| `gen_ai.tool.call.id` | `event_details.tool_call_id` (if present) | ❌ Not mapped |

---

## Fix

**`gen_ai.system` in `conversation_message` should be corrected to `gen_ai.provider.name`

Contributor guide

Open the contributing guide

Research direction

Review the app_log, conversation_message, custom_agents_log, and codex_log data streams through PRs #20205, #20208, #20209, and #20201. Compare each raw field with the ECS GenAI mappings listed here, including correcting gen_ai.system in conversation_message. Done means the requested provider, conversation, operation, tool, message, response, agent, and usage fields are mapped consistently across the four streams.

Written by the indexing model from the issue text.

Assessment

Tech stack
handlebars
Domain
observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.