elastic / elastic/elastic-serverless-forwarder

[Amazon Security Lake] Parquet File Support

Open
#506 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
38
Forks
50
PR merge metrics
No merged PRs in 30d

Description

**Describe the enhancement:**
Amazon Security Lake can store security telemetry from both AWS services and 3rd parties such as CrowdStrike, Zscaler and many more. Data is stored in Apache Parquet files. We have an existing integration to ingest this data from Security Lake and map it to ECS to ensure it can be leveraged within Elastic Security.

Our current integration relies upon Elastic Agent, but we see a strong demand amongst AWS customers to leverage Serverless Forwarder instead.

Can we add support to Servless Forwarder to read/decode Parquet files, to enable Amazon Security Lake customers to ingest data without the need for Elastic Agent?

PR's to highlight what was done on the Beats S3 input to add Parquet file support: https://github.com/elastic/beats/pull/35183 and https://github.com/elastic/beats/pull/35578

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the referenced Beats S3 input pull requests and the current Elastic Serverless Forwarder ingestion path. Determine how Parquet files from Amazon Security Lake should be read and decoded. Done means Security Lake data can be ingested through Serverless Forwarder without Elastic Agent and mapped for Elastic Security use.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cloud, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.