elastic / elastic/integrations
[Kubernetes]: GKE audit_logs missing ECS mappings
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Problem
Cloud support for `kubernetes.audit_logs` (#14554) ingests GKE via gcp-pubsub, but the pipeline never copies the GKE Cloud Logging / `protoPayload` fields into the same ECS fields EKS and AKS already get from native AuditEvent.
That breaks a single Kubernetes detection ruleset across providers. `new_terms` rules in particular cannot OR alternate fields — they need fields like `user.name`, `source.ip`, and `user_agent.original` populated the same way on all three.
## What we see
On GKE events these are empty / wrong:
| Field | EKS/AKS | GKE today |
|---|---|---|
| `user.name` | from `kubernetes.audit.user.username` | empty |
| `source.ip` | from `sourceIPs` | empty |
| `user_agent.original` | from `userAgent` | empty |
| `event.action` | from `verb` | empty |
| `orchestrator.resource.type` | from `objectRef.resource` (`secrets`, `pods`, …) | from `resource.type` → always `k8s_cluster` |
| `orchestrator.resource.name` | bare object name (e.g. `api-token`) | full `protoPayload.resourceName` path (`core/v1/namespaces/…/secrets/api-token`) |
| `orchestrator.namespace` | from `objectRef.namespace` (e.g. `my-ns`) | empty |
The data is on the event under `kubernetes.audit.protoPayload.*`:
- `authenticationInfo.principalEmail` (same as K8s usernames, e.g. `system:serviceaccount:…`)
- `requestMetadata.callerIp`
- `requestMetadata.callerSuppliedUserAgent`
- `methodName` (e.g. `io.k8s.core.v1.secrets.get`)
- `resourceName` (full resource path — needs parsing for name/namespace/kind)
## Ask
When `protoPayload` is present (GKE path), populate ECS fields so they match EKS/AKS:
1. `user.name` ← `protoPayload.authenticationInfo.principalEmail` (if `user.name` empty)
2. `source.ip` ← `protoPayload.requestMetadata.callerIp`
3. `user_agent.original` ← `protoPayload.requestMetadata.callerSuppliedUserAgent`
4. `event.action` ← verb parsed from `methodName` (last segment: `get`, `list`, `create`, …)
5. `orchestrator.resource.type` ← API resource kind from `methodName` / `resourceName` — **not** GCP monitored resource `k8s_cluster`
6. `orchestrator.resource.name` ← bare object name only (e.g. `api-token`), same as EKS/AKS
Today GKE sets this to the full `protoPayload.resourceName` path, which is not equivalent.
7. `orchestrator.namespace` ← namespace parsed from `resourceName` (e.g. `emu-secretharvest-nmap01`)
Today this field is empty on GKE.
Example parity we want for a secrets.get:
| Field | EKS/AKS today | GKE today | GKE target |
|---|---|---|---|
| `orchestrator.resource.name` | `api-token` | `core/v1/namespaces/…/secrets/api-token` | `api-token` |
| `orchestrator.namespace` | `my-ns` | *(empty)* | `my-ns` |
| `orchestrator.resource.type` | `secrets` | `k8s_cluster` | `secrets` |
## Why it matters
We want customers to have the option of using one Fleet Kubernetes ruleset for EKS, AKS, and GKE. Query ORs can resolve some KQL/ES|QL filters but they cannot fix `new_terms` field lists. Without this mapping, every UA/IP/user fingerprint rule is effectively EKS/AKS-only on this datastream.
Happy to share sample docs if useful.
Contributor guide
Research direction
Start by locating the kubernetes.audit_logs GKE/gcp-pubsub pipeline and compare its mappings with the existing EKS and AKS AuditEvent paths. Inspect sample events under kubernetes.audit.protoPayload and verify that the requested ECS and orchestrator fields match the stated EKS/AKS parity, including parsed resource names and namespaces.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, handlebars, kubernetes
- Domain
- backend, cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100