elastic / elastic/integrations

[openai] ECS GenAI Field Support for detections

Open
#20,744 1 comment 0 reactions 0 assignees View on GitHub
Integration:openai Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

**Related issue:**
- https://github.com/elastic/integrations/issues/20737

TRaDE is requesting that a new datastream be added to this integration for security detections.

## New datastream: `responses`

**Source:** OpenAI Responses API — `GET /v1/responses`
**Input:** CEL/HTTP poll returning stored per-request response objects with 30-day retention. Requires an API key with read scope (not the admin token). ZDR orgs will receive empty lists — the datastream will collect nothing but won't error.

## Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"openai"` | ❌ Not implemented — new datastream required |
| `gen_ai.request.model` | ✅ `model` field in response object | ❌ Not implemented |
| `gen_ai.response.model` | ✅ `model` field in response object | ❌ Not implemented |
| `gen_ai.request.max_tokens` | ✅ `max_output_tokens` field in response object | ❌ Not implemented |
| `gen_ai.request.temperature` | ✅ `temperature` field in response object | ❌ Not implemented |
| `gen_ai.request.seed` | ✅ `seed` field in response object (if set by client) | ❌ Not implemented |
| `gen_ai.operation.name` | ✅ Inferable from endpoint — always `"chat"` | ❌ Not implemented |
| `gen_ai.response.id` | ✅ `id` field in response object | ❌ Not implemented |
| `gen_ai.response.finish_reasons` | ✅ `status` / `incomplete_details` fields in response object | ❌ Not implemented |
| `gen_ai.usage.input_tokens` | ✅ `usage.input_tokens` field in response object | ❌ Not implemented |
| `gen_ai.usage.output_tokens` | ✅ `usage.output_tokens` field in response object | ❌ Not implemented |
| `gen_ai.usage.cache_read.input_tokens` | ✅ `usage.input_tokens_details.cached_tokens` field in response object | ❌ Not implemented |
| `gen_ai.system_instructions` | ✅ `instructions` field in response object | ❌ Not implemented |
| `gen_ai.input.messages` | ✅ Input array in response object | ❌ Not implemented |
| `gen_ai.output.messages` | ✅ Output array in response object | ❌ Not implemented |
| `gen_ai.output.type` | ✅ Inferable from output item `type` field | ❌ Not implemented |
| `gen_ai.tool.definitions` | ✅ `tools` array in response object | ❌ Not implemented |
| `gen_ai.tool.call.id` | ✅ Present in tool call output items | ❌ Not implemented |
| `gen_ai.tool.name` | ✅ Present in tool call output items | ❌ Not implemented |
| `gen_ai.tool.call.arguments` | ✅ Present in tool call output items | ❌ Not implemented |

## Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.tool.call.result` | ❌ Tool call results live in the next request's input, not in the stored response object | ❌ Not implemented |
| `gen_ai.conversation.id` (OTel) | ⚠️ No explicit conversation ID field — `previous_response_id` chains responses but no flat session ID exists | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ❌ Not stored in response object | ❌ Not implemented |
| `gen_ai.agent.id/name/description/version` | ❌ Not in Responses API — would require Assistants API | ❌ Not implemented |
| `gen_ai.token.type` | ❌ Aggregate concept, not applicable per-request | ❌ Not implemented |

## Thinking Block Collection — End-to-End Configuration Guide

**Provider config:** Set `summary: "auto"`, `"concise"`, or `"detailed"` on requests. Must be set at request time; stored responses reflect whatever was requested.

**What lands in raw data** (proposed `responses` datastream):
```json
output[]: [
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "..."}]},
{"type": "message", ...}
]
```

**Integration gap:** The proposed `responses` datastream does not exist yet. When built, it would need to extract `output[]` items where `type == "reasoning"` and map their `summary[]` content.

Contributor guide

Open the contributing guide

Research direction

Start with related issue 20737 and the OpenAI Responses API endpoint described here; verify the listed fields and edge cases on a development organization, including ZDR empty results. Done means a new responses datastream collects the available detection data, maps the feasible fields and reasoning summaries, and handles unavailable fields without errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
handlebars
Domain
api, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.