awslabs / awslabs/aws-lambda-redshift-loader

Dynamic tablename/config.json use case

Open
#104 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
595
Forks
161
PR merge metrics
No merged PRs in 30d

Description

I'm looking at a use case where I'd like to bring in CSVs into their own individual tables, created at/right before load time. This is for data submitted by internal users. It benefits us to isolate each CSV (relatively small in size) to it's own table. This table would be determined by the filename and the s3 directories in which the CSV is submitted to.

Option 1 is to load all these CSVs into a single, pre-created table, and then have a lambda run off of an SNS msg to create a table and move data to this table for each file. I think this gets a little tricky when considering concurrency.

Option 2 is to first run a custom lambda function that will create the end table we'd load into, and then insert into dynamodb a config setting for this table. This lambda function then would, at the end, move the s3 file into a location that would trigger the aws redshift lambda loader and use the newly pushed config.

I'm inclined to go with option 2, but I'm wondering if there is anything to be aware of when doing this. It seems like to me that I need to insert into dynamodb a new config that contains the tablename, s3 location, etc as a part of the pre-process lambda. Am I missing anything else?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.