elastic / elastic/beats

Allow for multiple outputs in filebeat

Open
#42,957 3 comments 1 reaction 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

I have some use-cases where we require the ability of specifying multiple outputs. For example:

- shipping logs to a topic on two different kafka clusters
- shipping logs to a kafka topic as well as writing them to a file

I'm aware, this has already been discussed in this [issue](https://github.com/elastic/beats/issues/28925) but was made clear at the time there were no plans on adding it. In our case, we definitely don't want to have multiple instances of filebeat running just a accomplish this as it's much cleaner and simpler to manage it via a single running application. Also, when it comes to monitoring each of these potential filebeat instances, each of them would require its own port for exposing the HTTP server to fetch metrics from it. Working in environments with stricter security requirements where ports need to be pre-approved and opened by another team on a central firewall, this makes releasing any new filbeat instance quickly very painful.

We do have other potential filebeat replacements that could use to allow for this feature (Logstash is out of the question unfortunately) although I would prefer using filebeat seeing it's already deployed on hosts.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by reading the prior discussion in issue #28925 and locating Filebeat's existing output configuration; done means one Filebeat instance can send events to multiple Kafka clusters and to Kafka plus a file.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.