awslabs / awslabs/amazon-redshift-utils

Replay Util: In current logic COPY replacements will always yield 0 replacements

Open
#589 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.8k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

in the first iteration of our tests to replay copy statements we saw that all copy statements were executed and failed.
when looking into the logs we see the the logs yilded/parsed 0 statements based on this code.
"Loaded {len(copy_replacements)} COPY replacements from {replacements_path}"

looking into the code it seems that this will always be the case because in the extract phase the csv file copy_replacements.csv is built from the file locations of the orioginal files - which in our case was not the case because we replayed the workloads in a different aws account and copied the files used by the copy commands from the original account (like the aws blog says) - and then the replay part makes condition on the next 2 cells (destination file, replacement iam role) so it always will find 0.

the only thing we needed to do to overcome this is to quickly after the replay part generates the replacements file - to inject instead our own corrected file with the new destination files and replacement iam rolw.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the extract and replay phases that generate and consume copy_replacements.csv, especially the log reporting the loaded replacement count. Reproduce the COPY replay path and verify that replacements using the replayed destination files and IAM roles are loaded and applied instead of yielding zero matches.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
databases, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.