elastic / elastic/package-spec

Add support for routing rules or pre-defined pipelines in input packages

Open
#566 6 comments 1 reaction 0 assignees View on GitHub
stale Team:Fleet
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

We want to create a new input package called `AWS Firehose` that streams logs (and metrics in the future) from Amazon Kinesis Data Firehose into Elastic Stack. This input package needs to reroute incoming logs/metrics to different data streams based on the pre-defined rerouting rules or pipelines.

For example: We would like to route the log entry to `aws.cloudtrail` dataset if the value of `aws.cloudwatch.log_stream` field has `CloudTrail` in there.
```
processors:
- reroute:
if: ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
dataset: aws.cloudtrail
namespace: default
```

Either supporting routing rules directly or supporting pre-defined pipelines will work for our use case.

cc @jsoriano @mrodm @tommyers-elastic

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.