aws-samples / aws-samples/cross-account-amazon-dynamodb-replication
ChangeDataCapture base64 encodes binary values
- Dominant language
- Python
- Stars
- 28
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
The dynamodb stream is emitting it's changes in json string format, with binary values being base64 encoded. However when performing the putitem command through boto, these strings are assumed to be un-encoded and go through an additional base64 encoding.
I was able to resolve this by transforming the binary objects into their bytes representations before sending them through boto
```
if event_name == 'REMOVE':
keys_to_delete = record['dynamodb']['Keys']
# Look for binary values b64 decode them back into bytes
for key in keys_to_delete:
for valType in keys_to_delete[key]:
if valType == 'B':
keys_to_delete[key][valType] = base64.b64decode(keys_to_delete[key][valType])
dynamodb.delete_item(TableName=target_ddb_name,Key=record['dynamodb']['Keys'])
else:
#print("Putting item")
item_to_put = record['dynamodb']['NewImage']
# Look for binary values b64 decode them back into bytes
for property in item_to_put:
for valType in item_to_put[property]:
if valType == 'B':
item_to_put[property][valType] = base64.b64decode(item_to_put[property][valType])
dynamodb.put_item(TableName=target_ddb_name,Item=item_to_put)
```
Contributor guide
Research direction
Start by locating the handler that reads DynamoDB stream records and calls boto's put_item and delete_item operations, then trace how binary attributes are represented after JSON parsing. Reproduce the replication with a binary key or value and verify that copied records and removals preserve the original binary data without an additional base64 encoding.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100