elastic / elastic/integrations
[gcp_vertexai] ECS GenAI Field Support for detections
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
**Related issue:**
- https://github.com/elastic/integrations/issues/20737
TRaDE is requesting that the following fields be added to this integration for security detections:
## Requested fields
| Field | Availability | Status |
|---|---|---|
| `gen_ai.provider.name` | ✅ Can be hardcoded as `"vertex_ai"` (the OTel/ECS standard value for this provider) | ❌ Not implemented — no `gen_ai.*` fields exist in this integration |
| `gen_ai.request.model` | ✅ `gcp.vertexai.prompt_response_logs.model` (e.g., `"publishers/google/models/gemini-2.5-pro"`) and `full_request.model` (full resource path); `gcp.labels.resource.model_user_id` (e.g., `"gemini-1.5-flash-002"`) in `metrics` datastream | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.model` |
| `gen_ai.response.model` | ✅ `gcp.vertexai.prompt_response_logs.full_response.model_version` (e.g., `"gemini-2.5-pro"`) | ❌ Not implemented |
| `gen_ai.request.max_tokens` | ✅ `full_request.generation_config.max_output_tokens` present in `prompt_response_logs` (e.g., `8192`) | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_request.generation_config.max_output_tokens` |
| `gen_ai.request.temperature` | ✅ `full_request.generation_config.temperature` present in `prompt_response_logs` | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_request.generation_config.temperature` |
| `gen_ai.request.seed` | ⚠️ Vertex AI supports `seed` in `generation_config` but it only appears in the log if explicitly set by the client; not in sample events | ❌ Not implemented |
| `gen_ai.request.stop_sequences` | ⚠️ Vertex AI supports `stop_sequences` in `generation_config` but only present if explicitly set; not in sample events | ❌ Not implemented |
| `gen_ai.operation.name` | ✅ `gcp.vertexai.prompt_response_logs.api_method` present (e.g., `"GenerateContent"`); also captured in `auditlogs` via `protoPayload.methodName` | ⚠️ Incorrect mapping — `api_method` is copied to `event.action` only; `gen_ai.operation.name` is never set |
| `gen_ai.usage.input_tokens` | ✅ `full_response.usage_metadata.prompt_token_count` in `prompt_response_logs`; aggregate per-model counts also available in `metrics` datastream via `publisher.online_serving.token_count` with `type=input` label | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_response.usage_metadata.prompt_token_count` |
| `gen_ai.usage.output_tokens` | ✅ `full_response.usage_metadata.candidates_token_count` in `prompt_response_logs`; also available in `metrics` with `type=output` label | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_response.usage_metadata.candidates_token_count` |
| `gen_ai.response.finish_reasons` | ✅ `full_response.candidates[].finish_reason` present in `prompt_response_logs` (e.g., `"STOP"`) | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_response.candidates[].finish_reason` |
| `gen_ai.input.messages` | ✅ `full_request.contents[]` with `role` and `parts[].text` fully captured in `prompt_response_logs` | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_request.contents` |
| `gen_ai.output.messages` | ✅ `full_response.candidates[].content` with `role` and `parts[].text` fully captured in `prompt_response_logs` | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_response.candidates` |
| `gen_ai.response.id` | ✅ `full_response.response_id` present in `prompt_response_logs` | ❌ Not implemented |
| `gen_ai.system_instructions` | ✅ `full_request.system_instruction.parts.text` present in `prompt_response_logs` schema and populated when system instructions are set | ❌ Not implemented — stored as `gcp.vertexai.prompt_response_logs.full_request.system_instruction` |
| `gen_ai.tool.definitions` | ⚠️ Vertex AI supports function/tool declarations in requests; would be captured in `full_request` when tools are used, but no schema fields or sample events cover this | ❌ Not implemented |
| `gen_ai.tool.call.id` | ⚠️ Present in `full_response.candidates[].content.parts[]` as `function_call` objects when tools are invoked; captured in logs but no schema fields cover it | ❌ Not implemented |
| `gen_ai.tool.name` | ⚠️ Present in `full_response.candidates[].content.parts[].function_call.name` when tools are invoked | ❌ Not implemented |
| `gen_ai.tool.type` | ✅ Not a discrete field; all Vertex AI tools are of type `function` and this is not explicitly logged | ❌ Not implemented |
| `gen_ai.tool.call.result` | ⚠️ Would appear in `full_request.contents[].parts[]` as `function_response` objects when tool results are returned to the model | ❌ Not implemented |
| `gen_ai.tool.call.arguments` | ⚠️ Present in `full_response.candidates[].content.parts[].function_call.args` when tool calls are made | ❌ Not implemented |
## Likely not possible (verify on development)
| Field | Availability | Status |
|---|---|---|
| `gen_ai.conversation.id` (OTel) | ❌ No conversation or session ID present in any Vertex AI log source | ❌ Not implemented |
| `gen_ai.token.type` | ❌ Not a discrete field; `candidates_tokens_details[].modality` and `prompt_tokens_details[].modality` indicate token modality (e.g., `"TEXT"`) but this is not equivalent to the ECS `token.type` concept | ❌ Not implemented |
| `gen_ai.output.type` | ❌ Not a discrete field in Vertex AI logs; content type is not separately logged | ❌ Not implemented |
## Thinking Block Collection — End-to-End Configuration Guide
**Provider config:**
- Enable prompt/response logging in the Vertex AI console
- Caller must enable thinking and set `thinking_summaries` on requests to get readable summaries
- `thoughts_token_count` appears regardless; summary content requires the opt-in
**What lands in raw data:**
```json
full_response.candidates[].content.parts[]: [
{"thought": true, "text": ""},
{"text": "..."}
]
full_response.usage_metadata.thoughts_token_count: 901
```
**Integration gap:** `thoughts_token_count` is already mapped. Thinking summary content in `parts[]` is not distinguished from text parts — would need to filter `parts[]` where `thought == true`.
Contributor guide
Research direction
Start by inspecting the gcp_vertexai integration and its existing prompt_response_logs, auditlogs, and metrics mappings. Compare the available Vertex AI fields with the requested ECS GenAI fields, then verify development samples for fields marked as uncertain or unavailable. Done means the supported mappings and thinking-block handling are represented consistently, with tests or sample validation covering the changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp
- Domain
- ai, cloud, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100