elastic / elastic/integrations

[Fortinet Fortigate] Split current dataset into multiple datasets

Open
#12,606 39 comments 2 reactions 0 assignees View on GitHub
Integration:fortinet_fortigate Team:Integration-Experience
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Hello,

Currently the integration for the Fortinet Fortigate has only one dataset, `fortinet_fortigate.log`, which means that all logs from fortigate will be stored in the same data stream, but the integration produces different types of logs like `traffic`, `utm` and `event`.

The goal for this is that this would allow you to have different retention per dataset, for example, you may be required to keep audit logs for a longer time than traffic logs, audit logs have the `fortinet.firewall.type` value as `event` and traffic logs have the same field as `traffic`.

Since the volume of `traffic` logs are higher, to keep only the audit logs you do not have many options, you basically need to create a custom transform to store those events in another index, I'm not sure that you can use a `reroute` processor to create a custom dataset/namespace since Fleet integrations permissions are pretty limited.

Now we have this:

- `fortinet_fortigate.log`: all logs from fortigate

But we should have something like this:

- `fortinet_fortigate.traffic`: traffic logs from fortigate
- `fortinet_fortigate.event`: event logs from fortigate
- `fortinet_fortigate.utm`: utm logs from fortigate
- `fortinet_fortigate.log`: new/unmapped logs from fortigate

We already have separated ingest pipelines for each `fortinet.firewall.type` value [here](https://github.com/elastic/integrations/blob/ed3158658d5f72af36d63d02aba612fd24f368e2/packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline/default.yml#L252-L268).

```
- pipeline:
name: '{{ IngestPipeline "event" }}'
if: ctx.fortinet?.firewall?.type == 'event'
- pipeline:
name: '{{ IngestPipeline "login" }}'
if: >-
if (ctx.message instanceof String) {
String normalizedMessage = ctx.message.toLowerCase();
return (normalizedMessage.contains('login') || normalizedMessage.contains('logged in'));
}
return false;
- pipeline:
name: '{{ IngestPipeline "traffic" }}'
if: ctx.fortinet?.firewall?.type == 'traffic'
- pipeline:
name: '{{ IngestPipeline "utm" }}'
if: ctx.fortinet?.firewall?.type == 'utm' || ctx.fortinet?.firewall?.type == 'dns'
```

I think this helps implement separated data streams, I'm not sure how to do that, but with some orientation I could send a PR.

What we need to do to create a new dataset?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.