Need to Configure AWS for Fluent Bit to Send Raw Logs to S3 for Spark History Server
- Dominant language
- Mustache
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
**Description:**
We are using the "aws-for-fluent-bit" application to send logs from an Amazon EKS cluster to an S3 bucket. Our current setup adds Kubernetes metadata (e.g., pod name, namespace, etc.) and additional fields to the logs, which is suitable for most use cases. However, we have a requirement to send logs in their raw format (as generated by the application) for a spark history server, without any added metadata or fields.
**The challenge we are facing is twofold:**
1. We need to send raw logs (without enrichment) to the S3 bucket for one application.
2. For all other applications, we want to continue sending logs with the current enriched format, including Kubernetes metadata.
**We are seeking guidance on how to configure the "aws-for-fluent-bit" application to:**
• Process logs as raw data and send them to S3.
• Continue sending enriched logs for all other applications.
**Request:**
• How can we configure Fluent Bit to handle logs differently based on the Kubernetes app, ensuring raw logs for one app and enriched logs for others?
• Are there best practices or specific configuration recommendations for this use case?
Contributor guide
Research direction
Start by reviewing the aws-for-fluent-bit configuration for Amazon EKS, focusing on routing logs by Kubernetes application and preserving raw output for the Spark History Server. No files or tests are named in the issue. Done means one application sends un-enriched logs to S3 while all other applications retain their Kubernetes metadata and additional fields.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes, spark
- Domain
- cloud, data-engineering, observability-sre
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100