awslabs / awslabs/amazon-redshift-utils
Replay Util: In current logic COPY replacements will always yield 0 replacements
- Dominant language
- Python
- Stars
- 2.8k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
in the first iteration of our tests to replay copy statements we saw that all copy statements were executed and failed.
when looking into the logs we see the the logs yilded/parsed 0 statements based on this code.
"Loaded {len(copy_replacements)} COPY replacements from {replacements_path}"
looking into the code it seems that this will always be the case because in the extract phase the csv file copy_replacements.csv is built from the file locations of the orioginal files - which in our case was not the case because we replayed the workloads in a different aws account and copied the files used by the copy commands from the original account (like the aws blog says) - and then the replay part makes condition on the next 2 cells (destination file, replacement iam role) so it always will find 0.
the only thing we needed to do to overcome this is to quickly after the replay part generates the replacements file - to inject instead our own corrected file with the new destination files and replacement iam rolw.
Contributor guide
Research direction
Start by tracing the extract and replay phases that generate and consume copy_replacements.csv, especially the log reporting the loaded replacement count. Reproduce the COPY replay path and verify that replacements using the replayed destination files and IAM roles are loaded and applied instead of yielding zero matches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- databases, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100