elastic / elastic/integrations

[azure_ai_foundry] ECS GenAI Field Support for detections

Open
#20,890 1 comment 0 reactions 0 assignees View on GitHub
Integration:azure_ai_foundry Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

**Related issues:**
- https://github.com/elastic/integrations/issues/20737
- https://github.com/elastic/integrations/issues/20739 (azure_openai — similar scope)

TRaDE is requesting that ECS \`gen_ai.*\` fields be added to this integration for security detections.

The \`azure_ai_foundry\` package has two data streams: \`logs\` (chat completions, content filtering, gateway, audit events) and \`metrics\` (aggregate token/request/latency metrics). The \`logs\` data stream contains rich per-request LLM telemetry and is the primary target for gen_ai field mapping.

## \`logs\` data stream — Requested fields

| Field | Availability | Status |
|---|---|---|
| \`gen_ai.provider.name\` | ✅ Inferable — provider is Azure AI Foundry | ❌ Not implemented |
| \`gen_ai.operation.name\` | ✅ \`azure.ai_foundry.operation_name\` present in source logs (e.g., \`"ChatCompletions"\`) | ❌ Not implemented |
| \`gen_ai.request.model\` | ✅ \`azure.ai_foundry.properties.model_name\` present in source logs | ❌ Not implemented |
| \`gen_ai.response.model\` | ✅ \`azure.ai_foundry.properties.backend_response_body.model\` present in source logs | ❌ Not implemented |
| \`gen_ai.response.id\` | ✅ \`azure.ai_foundry.properties.backend_response_body.id\` present in source logs | ❌ Not implemented |
| \`gen_ai.usage.input_tokens\` | ✅ \`azure.ai_foundry.properties.backend_response_body.usage.input_tokens\` present in source logs | ❌ Not implemented |
| \`gen_ai.usage.output_tokens\` | ✅ \`azure.ai_foundry.properties.backend_response_body.usage.output_tokens\` present in source logs | ❌ Not implemented |
| \`gen_ai.input.messages\` | ✅ \`properties.backend_request_body.messages[].role\` and \`.content\` present in source logs | ❌ Not implemented — data exists but not normalized into ECS \`flattened\` structure |
| \`gen_ai.output.messages\` | ✅ \`properties.backend_response_body.choices[].message.role\` and \`.content\` present in source logs | ❌ Not implemented — data exists but not normalized into ECS \`flattened\` structure |
| \`gen_ai.response.finish_reasons\` | ✅ \`properties.backend_response_body.choices[].finish_reason\` present in source logs | ❌ Not implemented |

## \`metrics\` data stream — Requested fields

The metrics data stream captures aggregate per-model statistics (total tokens, request counts, latency averages). Per-request context (messages, finish reasons, response IDs) is not available here.

| Field | Availability | Status |
|---|---|---|
| \`gen_ai.provider.name\` | ✅ Inferable — provider is Azure AI Foundry | ❌ Not implemented |
| \`gen_ai.request.model\` | ⚠️ Model name available as a metric dimension — verify exact field name during development | ❌ Not implemented |

## Likely not possible (verify on development)

| Field | Availability | Status |
|---|---|---|
| \`gen_ai.request.temperature\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.request.max_tokens\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.request.top_p\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.request.seed\` | ❌ Not logged in Azure AI Foundry APIM gateway logs | ❌ Not implemented |
| \`gen_ai.agent.*\` | ❌ No agent orchestration telemetry in this integration | ❌ Not implemented |
| \`gen_ai.tool.*\` | ❌ No tool call telemetry in this integration | ❌ Not implemented |
| \`gen_ai.conversation.id\` | ❌ Not present in source logs | ❌ Not implemented |
| \`gen_ai.system_instructions\` | ❌ System prompt not broken out separately from messages array | ❌ Not implemented |

## Summary of changes needed

### \`logs\` data stream
- Add ECS \`gen_ai.*\` field definitions to \`fields/fields.yml\`
- Add ingest pipeline processors to set \`gen_ai.provider.name\`, \`gen_ai.operation.name\`, copy model fields, copy usage token counts, copy \`response.id\`, collect \`finish_reasons\` from the choices array, and normalize \`input.messages\` / \`output.messages\` into ECS flattened structure
- Update \`sample_event.json\` to include \`gen_ai.*\` fields

### \`metrics\` data stream
- Add \`gen_ai.provider.name\` and \`gen_ai.request.model\` field definitions and pipeline processors
- Update \`sample_event.json\` to include \`gen_ai.*\` fields

Contributor guide

Open the contributing guide

Research direction

Start by inspecting fields/fields.yml, the logs and metrics ingest pipelines, and each data stream's sample_event.json; compare the listed source paths with existing ECS mappings. Confirm the metrics model dimension and unavailable fields during development, then update definitions, processors, and samples so the requested gen_ai fields are present in both streams.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.