elastic / elastic/integrations

[Kubernetes]: GKE audit_logs missing ECS mappings

Open
#20,861 1 comment 0 reactions 0 assignees View on GitHub
enhancement Integration:kubernetes Team:obs-ds-hosted-services Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Problem

Cloud support for `kubernetes.audit_logs` (#14554) ingests GKE via gcp-pubsub, but the pipeline never copies the GKE Cloud Logging / `protoPayload` fields into the same ECS fields EKS and AKS already get from native AuditEvent.

That breaks a single Kubernetes detection ruleset across providers. `new_terms` rules in particular cannot OR alternate fields — they need fields like `user.name`, `source.ip`, and `user_agent.original` populated the same way on all three.

## What we see

On GKE events these are empty / wrong:

| Field | EKS/AKS | GKE today |
|---|---|---|
| `user.name` | from `kubernetes.audit.user.username` | empty |
| `source.ip` | from `sourceIPs` | empty |
| `user_agent.original` | from `userAgent` | empty |
| `event.action` | from `verb` | empty |
| `orchestrator.resource.type` | from `objectRef.resource` (`secrets`, `pods`, …) | from `resource.type` → always `k8s_cluster` |
| `orchestrator.resource.name` | bare object name (e.g. `api-token`) | full `protoPayload.resourceName` path (`core/v1/namespaces/…/secrets/api-token`) |
| `orchestrator.namespace` | from `objectRef.namespace` (e.g. `my-ns`) | empty |

The data is on the event under `kubernetes.audit.protoPayload.*`:

- `authenticationInfo.principalEmail` (same as K8s usernames, e.g. `system:serviceaccount:…`)
- `requestMetadata.callerIp`
- `requestMetadata.callerSuppliedUserAgent`
- `methodName` (e.g. `io.k8s.core.v1.secrets.get`)
- `resourceName` (full resource path — needs parsing for name/namespace/kind)

## Ask

When `protoPayload` is present (GKE path), populate ECS fields so they match EKS/AKS:

1. `user.name` ← `protoPayload.authenticationInfo.principalEmail` (if `user.name` empty)
2. `source.ip` ← `protoPayload.requestMetadata.callerIp`
3. `user_agent.original` ← `protoPayload.requestMetadata.callerSuppliedUserAgent`
4. `event.action` ← verb parsed from `methodName` (last segment: `get`, `list`, `create`, …)
5. `orchestrator.resource.type` ← API resource kind from `methodName` / `resourceName` — **not** GCP monitored resource `k8s_cluster`
6. `orchestrator.resource.name` ← bare object name only (e.g. `api-token`), same as EKS/AKS
Today GKE sets this to the full `protoPayload.resourceName` path, which is not equivalent.
7. `orchestrator.namespace` ← namespace parsed from `resourceName` (e.g. `emu-secretharvest-nmap01`)
Today this field is empty on GKE.

Example parity we want for a secrets.get:

| Field | EKS/AKS today | GKE today | GKE target |
|---|---|---|---|
| `orchestrator.resource.name` | `api-token` | `core/v1/namespaces/…/secrets/api-token` | `api-token` |
| `orchestrator.namespace` | `my-ns` | *(empty)* | `my-ns` |
| `orchestrator.resource.type` | `secrets` | `k8s_cluster` | `secrets` |

## Why it matters

We want customers to have the option of using one Fleet Kubernetes ruleset for EKS, AKS, and GKE. Query ORs can resolve some KQL/ES|QL filters but they cannot fix `new_terms` field lists. Without this mapping, every UA/IP/user fingerprint rule is effectively EKS/AKS-only on this datastream.

Happy to share sample docs if useful.

Contributor guide

Open the contributing guide

Research direction

Start by locating the kubernetes.audit_logs GKE/gcp-pubsub pipeline and compare its mappings with the existing EKS and AKS AuditEvent paths. Inspect sample events under kubernetes.audit.protoPayload and verify that the requested ECS and orchestrator fields match the stated EKS/AKS parity, including parsed resource names and namespaces.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, handlebars, kubernetes
Domain
backend, cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.