awslabs / awslabs/amazon-redshift-utils

[Question] Not providing the AWS access key

Open
#534 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.8k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

Hello,

I am trying to use the `unload-copy` module and I can see that besides `CONFIG_FILE` and `AWS_REGION`, also `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are needed. I assume the access key is needed to access KMS for decrypting the secrets, however, I can see that if I do not provide them then we have a problem here:

https://github.com/awslabs/amazon-redshift-utils/blob/ba5bcb588fa3d684e663aea0f2808b7e5c29dd81/src/UnloadCopyUtility/global_config.py#L154-L155

The thrown exception is due the fact that `boto3` needs these parameters but my question is why is EC2 client needed? Is this for checking the region only?

I am asking these because my use case is to use this utility in an ECS task to migrate some data between 2 Redshift databases. I need to inject the secrets through environment variables from SSM, hence they are in plaintext in the container. I tried to generate the `CONFIG_FILE` internally and further provide it to `bin/run-unload-copy-utility.sh`. In this case there is no reason for storing the secrets encrypted in the config file since this is handled by SSM. Therefore I do not need the IAM access key but I am afraid that the current implementation forces me to do the opposite. Is there a way to use the utility as I specified - without the access key?

Thanks in advance!

Contributor guide

Open the contributing guide

Research direction

Start with src/UnloadCopyUtility/global_config.py at the referenced lines and trace how CONFIG_FILE, AWS_REGION, AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY are loaded. Review bin/run-unload-copy-utility.sh and the unload-copy configuration path for the ECS/SSM use case. Done means the utility's required credential behavior is clearly established or the requested credential-free configuration path is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, databases, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.