elastic / elastic/beats

Add s3 output to filebeat

Open
#18,158 16 comments 27 reactions 0 assignees View on GitHub
enhancement Team:Integrations
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

It would be very helpful to allow filebeat to output to s3 directly.

Currently, if one wants to store logs on s3, logstash is required.

When using a service like aws elastic beanstalk, it is very handy to push logs to s3 for persistence. I can imagine 100 other usecases but this one is the current one that would have simplified my life.

The only alternative is rather complicated - you have to configure the aws ECS agent to support gelf logging and then use gelf logging to logstash. And then push the logs to s3 via logstash output.

Major downsite: There is a race-condition with this approach where you will lose the initial logs from containers that start before the logstash container.

One big downside: you cannot use docker logs for quick inspection anymore because aws doesn't offer dual logger output. The json-file supported by file beat would work out of the box here. It would certainly be easier to use filebeat when just getting started

Another issue is that you must run logstash on each application instance plus the ones you need for ingestion into elasticsearch.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by surveying existing Filebeat output implementations and clarifying the required S3 behavior, including how it should support the described container logging workflow. Done means Filebeat can send logs directly to S3 without requiring Logstash.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, go
Domain
backend, cloud, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.