elastic / elastic/integrations
[Spike] Determine best field for setting log source type across security integrations
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 182
Description
## Description
Users want to quickly determine what type of log source produced an event when searching across `logs-*`. Current fields don't offer a consistently populated answer.
`observer.type` is a potential candidate but is inconsistently populated across sources and its ECS definition might imply a narrower scope (external network/security appliances) than the use case requires. No single field satisfies the need reliably.
## User context
A customer migrating from a legacy SIEM has flagged this, with their previous platform having surfaced source type and name on every event (ie ` @ `).
Confirmed affected integrations include:
- azure (Gateway, Firewall, Application, Keyvault)
- system (auth)
- palo_alto_networks
- checkpoint
Without a reliable equivalent in Elastic, logs from a Palo Alto firewall, Checkpoint firewall and Windows host aren't easily distinguishable in `logs-*` search results:
```
I am searching in logs-* for a specific source IP address and get data from a Palo Alto Firewall, a CheckPoint Firewall, and the logs of one Windows Host.
In our old system, we were able to see the source of the log in every log in the following format: " @ ".
We are currently trying to use (or misuse) the field observer.type (https://www.elastic.co/docs/reference/ecs/ecs-observer#field-observer-type) for a categorization of the log source type.
Official description:
"The type of the observer the data is coming from.
There is no predefined list of observer types. Some examples are forwarder, firewall, ids, ips, proxy, poller, sensor, APM server.
type: keyword
example: firewall"
Unfortunately, the field is not filled automatically in every log - even not in standard parsers.
Our first intuitive choice would have been "data_stream.dataset", but this was sometimes filled with the name of the custom parser which would not be understood by the analysts.
Helpful would be a dashboard directly in the Security module (like "Explore" > "Logs") with an overview of all the connected sources and their system types, e.g.
Observer type: firewall
Observer product: Prisma Access
```
If `observer.type` (or a similar field) was reliably populated with clean values like `firewall`, `endpoint`, `server` etc. across all integrations, analysts could filter ("show me only logs where source type = `firewall`") and add a dashboard breakdown of event volume by source type without custom ingest pipelines.
## Questions
1. Is `observer.type` the right field for all log source types, or is it intended only for external network/security devices? How should OS-level sources and cloud/API-based sources be classified? The ECS definition of `observer.type` describes it as "_a special network, security or application device used to detect, observe or create network, security or application-related events_".
2. Is there a consistent recommended set of values for `observer.type` that could be defined and enforced during integration review?
3. Could a multi-field pattern help power the filtering use case? (ie `observer.type` = `firewall`, `observer.vendor` = `Palo Alto Networks`, `observer.product` = `Prisma Access`)?
4. Could the downstream ask (a source inventory dashboard in Explore > Logs) be raised as a feature request at the Kibana level, and is field consistency a prerequisite for this?
5. What integrations populate `observer.type` today?
## Requested outcome
Engineering/Product to assess the right field strategy and surface a recommendation that can inform some combination of a follow-on enhancement issue, proposal to ECS, or Kibana feature request.
Contributor guide
Assessment
This issue has not been assessed yet.