elastic / elastic/integrations
[azure_ai_foundry] ECS GenAI Field Support for detections
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
**Related issues:**
- https://github.com/elastic/integrations/issues/20737
- https://github.com/elastic/integrations/issues/20739 (azure_openai — similar scope)
TRaDE is requesting that ECS \`gen_ai.*\` fields be added to this integration for security detections.
The \`azure_ai_foundry\` package has two data streams: \`logs\` (chat completions, content filtering, gateway, audit events) and \`metrics\` (aggregate token/request/latency metrics). The \`logs\` data stream contains rich per-request LLM telemetry and is the primary target for gen_ai field mapping.
## \`logs\` data stream — Requested fields
| Field | Availability | Status |
|---|---|---|
| \`gen_ai.provider.name\` | ✅ Inferable — provider is Azure AI Foundry | ❌ Not implemented |
| \`gen_ai.operation.name\` | ✅ \`azure.ai_foundry.operation_name\` present in source logs (e.g., \`"ChatCompletions"\`) | ❌ Not implemented |
| \`gen_ai.request.model\` | ✅ \`azure.ai_foundry.properties.model_name\` present in source logs | ❌ Not implemented |
| \`gen_ai.response.model\` | ✅ \`azure.ai_foundry.properties.backend_response_body.model\` present in source logs | ❌ Not implemented |
| \`gen_ai.response.id\` | ✅ \`azure.ai_foundry.properties.backend_response_body.id\` present in source logs | ❌ Not implemented |
| \`gen_ai.usage.input_tokens\` | ✅ \`azure.ai_foundry.properties.backend_response_body.usage.input_tokens\` present in source logs | ❌ Not implemented |
| \`gen_ai.usage.output_tokens\` | ✅ \`azure.ai_foundry.properties.backend_response_body.usage.output_tokens\` present in source logs | ❌ Not implemented |
| \`gen_ai.input.messages\` | ✅ \`properties.backend_request_body.messages[].role\` and \`.content\` present in source logs | ❌ Not implemented — data exists but not normalized into ECS \`flattened\` structure |
| \`gen_ai.output.messages\` | ✅ \`properties.backend_response_body.choices[].message.role\` and \`.content\` present in source logs | ❌ Not implemented — data exists but not normalized into ECS \`flattened\` structure |
| \`gen_ai.response.finish_reasons\` | ✅ \`properties.backend_response_body.choices[].finish_reason\` present in source logs | ❌ Not implemented |
## \`metrics\` data stream — Requested fields
The metrics data stream captures aggregate per-model statistics (total tokens, request counts, latency averages). Per-request context (messages, finish reasons, response IDs) is not available here.
| Field | Availability | Status |
|---|---|---|
| \`gen_ai.provider.name\` | ✅ Inferable — provider is Azure AI Foundry | ❌ Not implemented |
| \`gen_ai.request.model\` | ⚠️ Model name available as a metric dimension — verify exact field name during development | ❌ Not implemented |
## Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| \`gen_ai.request.temperature\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.request.max_tokens\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.request.top_p\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.request.seed\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.agent.*\` | ❌ No agent orchestration telemetry in this integration | ❌ Not implemented |
| \`gen_ai.tool.*\` | ❌ No tool call telemetry in this integration | ❌ Not implemented |
| \`gen_ai.conversation.id\` | ❌ Not present in source logs | ❌ Not implemented |
| \`gen_ai.system_instructions\` | ❌ System prompt not broken out separately from messages array | ❌ Not implemented |
## Summary of changes needed
### \`logs\` data stream
- Add ECS \`gen_ai.*\` field definitions to \`fields/fields.yml\`
- Add ingest pipeline processors to set \`gen_ai.provider.name\`, \`gen_ai.operation.name\`, copy model fields, copy usage token counts, copy \`response.id\`, collect \`finish_reasons\` from the choices array, and normalize \`input.messages\` / \`output.messages\` into ECS flattened structure
- Update \`sample_event.json\` to include \`gen_ai.*\` fields
### \`metrics\` data stream
- Add \`gen_ai.provider.name\` and \`gen_ai.request.model\` field definitions and pipeline processors
- Update \`sample_event.json\` to include \`gen_ai.*\` fields
Contributor guide
Research direction
Start by inspecting fields/fields.yml, the logs and metrics ingest pipelines, and each data stream's sample_event.json; compare the listed source paths with existing ECS mappings. Confirm the metrics model dimension and unavailable fields during development, then update definitions, processors, and samples so the requested gen_ai fields are present in both streams.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100