aws / aws/eks-charts

Need to Configure AWS for Fluent Bit to Send Raw Logs to S3 for Spark History Server

Open
#1,156 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Mustache
Stars
1.3k
Forks
1.1k
Avg merge
22m
Merged PRs (30d)
5

Description

**Description:**
We are using the "aws-for-fluent-bit" application to send logs from an Amazon EKS cluster to an S3 bucket. Our current setup adds Kubernetes metadata (e.g., pod name, namespace, etc.) and additional fields to the logs, which is suitable for most use cases. However, we have a requirement to send logs in their raw format (as generated by the application) for a spark history server, without any added metadata or fields.

**The challenge we are facing is twofold:**
1. We need to send raw logs (without enrichment) to the S3 bucket for one application.
2. For all other applications, we want to continue sending logs with the current enriched format, including Kubernetes metadata.

**We are seeking guidance on how to configure the "aws-for-fluent-bit" application to:**
• Process logs as raw data and send them to S3.
• Continue sending enriched logs for all other applications.

**Request:**
• How can we configure Fluent Bit to handle logs differently based on the Kubernetes app, ensuring raw logs for one app and enriched logs for others?
• Are there best practices or specific configuration recommendations for this use case?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the aws-for-fluent-bit configuration for Amazon EKS, focusing on routing logs by Kubernetes application and preserving raw output for the Spark History Server. No files or tests are named in the issue. Done means one application sends un-enriched logs to S3 while all other applications retain their Kubernetes metadata and additional fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, spark
Domain
cloud, data-engineering, observability-sre
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.