elastic / elastic/integrations
[openai] ECS GenAI Field Support for detections
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
**Related issue:**
- https://github.com/elastic/integrations/issues/20737
TRaDE is requesting that a new datastream be added to this integration for security detections.
## New datastream: `responses`
**Source:** OpenAI Responses API — `GET /v1/responses`
**Input:** CEL/HTTP poll returning stored per-request response objects with 30-day retention. Requires an API key with read scope (not the admin token). ZDR orgs will receive empty lists — the datastream will collect nothing but won't error.
## Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"openai"` | ❌ Not implemented — new datastream required |
| `gen_ai.request.model` | ✅ `model` field in response object | ❌ Not implemented |
| `gen_ai.response.model` | ✅ `model` field in response object | ❌ Not implemented |
| `gen_ai.request.max_tokens` | ✅ `max_output_tokens` field in response object | ❌ Not implemented |
| `gen_ai.request.temperature` | ✅ `temperature` field in response object | ❌ Not implemented |
| `gen_ai.request.seed` | ✅ `seed` field in response object (if set by client) | ❌ Not implemented |
| `gen_ai.operation.name` | ✅ Inferable from endpoint — always `"chat"` | ❌ Not implemented |
| `gen_ai.response.id` | ✅ `id` field in response object | ❌ Not implemented |
| `gen_ai.response.finish_reasons` | ✅ `status` / `incomplete_details` fields in response object | ❌ Not implemented |
| `gen_ai.usage.input_tokens` | ✅ `usage.input_tokens` field in response object | ❌ Not implemented |
| `gen_ai.usage.output_tokens` | ✅ `usage.output_tokens` field in response object | ❌ Not implemented |
| `gen_ai.usage.cache_read.input_tokens` | ✅ `usage.input_tokens_details.cached_tokens` field in response object | ❌ Not implemented |
| `gen_ai.system_instructions` | ✅ `instructions` field in response object | ❌ Not implemented |
| `gen_ai.input.messages` | ✅ Input array in response object | ❌ Not implemented |
| `gen_ai.output.messages` | ✅ Output array in response object | ❌ Not implemented |
| `gen_ai.output.type` | ✅ Inferable from output item `type` field | ❌ Not implemented |
| `gen_ai.tool.definitions` | ✅ `tools` array in response object | ❌ Not implemented |
| `gen_ai.tool.call.id` | ✅ Present in tool call output items | ❌ Not implemented |
| `gen_ai.tool.name` | ✅ Present in tool call output items | ❌ Not implemented |
| `gen_ai.tool.call.arguments` | ✅ Present in tool call output items | ❌ Not implemented |
## Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.tool.call.result` | ❌ Tool call results live in the next request's input, not in the stored response object | ❌ Not implemented |
| `gen_ai.conversation.id` (OTel) | ⚠️ No explicit conversation ID field — `previous_response_id` chains responses but no flat session ID exists | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ❌ Not stored in response object | ❌ Not implemented |
| `gen_ai.agent.id/name/description/version` | ❌ Not in Responses API — would require Assistants API | ❌ Not implemented |
| `gen_ai.token.type` | ❌ Aggregate concept, not applicable per-request | ❌ Not implemented |
## Thinking Block Collection — End-to-End Configuration Guide
**Provider config:** Set `summary: "auto"`, `"concise"`, or `"detailed"` on requests. Must be set at request time; stored responses reflect whatever was requested.
**What lands in raw data** (proposed `responses` datastream):
```json
output[]: [
{"type": "reasoning", "summary": [{"type": "summary_text", "text": "..."}]},
{"type": "message", ...}
]
```
**Integration gap:** The proposed `responses` datastream does not exist yet. When built, it would need to extract `output[]` items where `type == "reasoning"` and map their `summary[]` content.
Contributor guide
Research direction
Start with related issue 20737 and the OpenAI Responses API endpoint described here; verify the listed fields and edge cases on a development organization, including ZDR empty results. Done means a new responses datastream collects the available detection data, maps the feasible fields and reasoning summaries, and handles unavailable fields without errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- handlebars
- Domain
- api, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100