Allow for multiple outputs in filebeat
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
I have some use-cases where we require the ability of specifying multiple outputs. For example:
- shipping logs to a topic on two different kafka clusters
- shipping logs to a kafka topic as well as writing them to a file
I'm aware, this has already been discussed in this [issue](https://github.com/elastic/beats/issues/28925) but was made clear at the time there were no plans on adding it. In our case, we definitely don't want to have multiple instances of filebeat running just a accomplish this as it's much cleaner and simpler to manage it via a single running application. Also, when it comes to monitoring each of these potential filebeat instances, each of them would require its own port for exposing the HTTP server to fetch metrics from it. Working in environments with stricter security requirements where ports need to be pre-approved and opened by another team on a central firewall, this makes releasing any new filbeat instance quickly very painful.
We do have other potential filebeat replacements that could use to allow for this feature (Logstash is out of the question unfortunately) although I would prefer using filebeat seeing it's already deployed on hosts.
Contributor guide
Research direction
The issue names no files or tests. Start by reading the prior discussion in issue #28925 and locating Filebeat's existing output configuration; done means one Filebeat instance can send events to multiple Kafka clusters and to Kafka plus a file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100