aws-samples / aws-samples/cross-account-amazon-dynamodb-replication

ChangeDataCapture base64 encodes binary values

Open
#10 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
28
Forks
25
PR merge metrics
No merged PRs in 30d

Description

The dynamodb stream is emitting it's changes in json string format, with binary values being base64 encoded. However when performing the putitem command through boto, these strings are assumed to be un-encoded and go through an additional base64 encoding.

I was able to resolve this by transforming the binary objects into their bytes representations before sending them through boto
```
if event_name == 'REMOVE':
keys_to_delete = record['dynamodb']['Keys']

# Look for binary values b64 decode them back into bytes
for key in keys_to_delete:
for valType in keys_to_delete[key]:
if valType == 'B':
keys_to_delete[key][valType] = base64.b64decode(keys_to_delete[key][valType])

dynamodb.delete_item(TableName=target_ddb_name,Key=record['dynamodb']['Keys'])
else:
#print("Putting item")
item_to_put = record['dynamodb']['NewImage']

# Look for binary values b64 decode them back into bytes
for property in item_to_put:
for valType in item_to_put[property]:
if valType == 'B':
item_to_put[property][valType] = base64.b64decode(item_to_put[property][valType])
dynamodb.put_item(TableName=target_ddb_name,Item=item_to_put)
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the handler that reads DynamoDB stream records and calls boto's put_item and delete_item operations, then trace how binary attributes are represented after JSON parsing. Reproduce the replication with a binary key or value and verify that copied records and removals preserve the original binary data without an additional base64 encoding.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.