elastic / elastic/integrations
[Fortinet Fortigate] Split current dataset into multiple datasets
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
Hello,
Currently the integration for the Fortinet Fortigate has only one dataset, `fortinet_fortigate.log`, which means that all logs from fortigate will be stored in the same data stream, but the integration produces different types of logs like `traffic`, `utm` and `event`.
The goal for this is that this would allow you to have different retention per dataset, for example, you may be required to keep audit logs for a longer time than traffic logs, audit logs have the `fortinet.firewall.type` value as `event` and traffic logs have the same field as `traffic`.
Since the volume of `traffic` logs are higher, to keep only the audit logs you do not have many options, you basically need to create a custom transform to store those events in another index, I'm not sure that you can use a `reroute` processor to create a custom dataset/namespace since Fleet integrations permissions are pretty limited.
Now we have this:
- `fortinet_fortigate.log`: all logs from fortigate
But we should have something like this:
- `fortinet_fortigate.traffic`: traffic logs from fortigate
- `fortinet_fortigate.event`: event logs from fortigate
- `fortinet_fortigate.utm`: utm logs from fortigate
- `fortinet_fortigate.log`: new/unmapped logs from fortigate
We already have separated ingest pipelines for each `fortinet.firewall.type` value [here](https://github.com/elastic/integrations/blob/ed3158658d5f72af36d63d02aba612fd24f368e2/packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L252-L268).
```
- pipeline:
name: '{{ IngestPipeline "event" }}'
if: ctx.fortinet?.firewall?.type == 'event'
- pipeline:
name: '{{ IngestPipeline "login" }}'
if: >-
if (ctx.message instanceof String) {
String normalizedMessage = ctx.message.toLowerCase();
return (normalizedMessage.contains('login') || normalizedMessage.contains('logged in'));
}
return false;
- pipeline:
name: '{{ IngestPipeline "traffic" }}'
if: ctx.fortinet?.firewall?.type == 'traffic'
- pipeline:
name: '{{ IngestPipeline "utm" }}'
if: ctx.fortinet?.firewall?.type == 'utm' || ctx.fortinet?.firewall?.type == 'dns'
```
I think this helps implement separated data streams, I'm not sure how to do that, but with some orientation I could send a PR.
What we need to do to create a new dataset?
Contributor guide
Assessment
This issue has not been assessed yet.