awslabs / awslabs/aws-lambda-redshift-loader
Dynamic tablename/config.json use case
- Dominant language
- JavaScript
- Stars
- 595
- Forks
- 161
- PR merge metrics
- No merged PRs in 30d
Description
I'm looking at a use case where I'd like to bring in CSVs into their own individual tables, created at/right before load time. This is for data submitted by internal users. It benefits us to isolate each CSV (relatively small in size) to it's own table. This table would be determined by the filename and the s3 directories in which the CSV is submitted to.
Option 1 is to load all these CSVs into a single, pre-created table, and then have a lambda run off of an SNS msg to create a table and move data to this table for each file. I think this gets a little tricky when considering concurrency.
Option 2 is to first run a custom lambda function that will create the end table we'd load into, and then insert into dynamodb a config setting for this table. This lambda function then would, at the end, move the s3 file into a location that would trigger the aws redshift lambda loader and use the newly pushed config.
I'm inclined to go with option 2, but I'm wondering if there is anything to be aware of when doing this. It seems like to me that I need to insert into dynamodb a new config that contains the tablename, s3 location, etc as a part of the pre-process lambda. Am I missing anything else?
Contributor guide
Assessment
This issue has not been assessed yet.