airbnb / airbnb/streamalert

Feature: Verify/Support AWS Data Pipeline error logs

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
help wanted logs
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

**Background**

AWS Data Pipelines can log errors to an S3 bucket:

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-error-logs.html

**Goal**

- Identify the log format
- Verify if StreamAlert's existing S3 support does support this use-case
- If yes, update the documentation and close this issue
- If not, put up a PR to add support (schema / parser) if feasible

Contributor guide

Open the contributing guide

Research direction

Start with the AWS Data Pipeline error-log documentation linked in the issue and inspect StreamAlert's existing S3 support. Determine whether the log format is already supported; done means either documenting the verified support or proposing a PR for the schema/parser if feasible.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.