elastic / elastic/elastic-serverless-forwarder
[Amazon Security Lake] Parquet File Support
- Dominant language
- Python
- Stars
- 38
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the enhancement:**
Amazon Security Lake can store security telemetry from both AWS services and 3rd parties such as CrowdStrike, Zscaler and many more. Data is stored in Apache Parquet files. We have an existing integration to ingest this data from Security Lake and map it to ECS to ensure it can be leveraged within Elastic Security.
Our current integration relies upon Elastic Agent, but we see a strong demand amongst AWS customers to leverage Serverless Forwarder instead.
Can we add support to Servless Forwarder to read/decode Parquet files, to enable Amazon Security Lake customers to ingest data without the need for Elastic Agent?
PR's to highlight what was done on the Beats S3 input to add Parquet file support: https://github.com/elastic/beats/pull/35183 and https://github.com/elastic/beats/pull/35578
Contributor guide
Research direction
Start by reviewing the referenced Beats S3 input pull requests and the current Elastic Serverless Forwarder ingestion path. Determine how Parquet files from Amazon Security Lake should be read and decoded. Done means Security Lake data can be ingested through Serverless Forwarder without Elastic Agent and mapped for Elastic Security use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100