awslabs / awslabs/lambda-streams-to-firehose

Filtering by event type

Open
#59 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
278
Forks
90
PR merge metrics
No merged PRs in 30d

Description

I'm in the process of setting up DynamoDB archiving and was using the guide here:
https://aws.amazon.com/blogs/database/automatically-archive-items-to-s3-using-dynamodb-time-to-live-with-aws-lambda-and-amazon-kinesis-firehose/

That referred me to this repository for pushing my TTL'd data into Firehose from Lambda. That works great!
However, I'm getting every change made to dynamo rather than just the TTL'd docs. There is some verbage in the link I provided above that indicates I can check for the event name, etc..

> In this example, you archive all item changes to S3. If you wanted to archive only the items deleted by TTL, you could archive only the records where eventName is REMOVE and userIdentity contains PrincipalID equal to dynamodb.amazonaws.com.

Can you guys give me a snippet or some guidance on how to do this? I'm under the assumption I need to make a tweak to index file to handle this case.

Thanks!

Update:
I ended up just updating ln 467 to check for removal and principalId as follows:
```
if (record.eventName == "REMOVE" && record.userIdentity &&
record.userIdentity.principalId == "dynamodb.amazonaws.com") {

// dynamo update stream record
var data = exports.createDynamoDataItem(record);
```
Would you guys have a better suggestion?

Contributor guide

Open the contributing guide

Research direction

Start in index.js around line 467, where the issue identifies the DynamoDB stream record handling and calls createDynamoDataItem. Review the existing event filtering and verify that the forwarding behavior matches the requested TTL-removal criteria; done means non-TTL changes are excluded without breaking the existing Firehose path.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript
Domain
cloud
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.