aws / aws/aws-cdk

(aws-cdk-deployment): (AWS CDK deployment modfies binary data files that are non utf-8 encoded before uploading them to S3)

Open
#22,134 2 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-s3 effort/medium feature-request p2
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

### Describe the bug

I was trying to use https://docs.aws.amazon.com/cdk/api/v1/docs/aws-s3-deployment-readme.html to upload a binary file. My specific use case is to

1. Create a google protobuf file descriptor set file. https://googleapis.dev/python/protobuf/3.11.2/google/protobuf/descriptor.html locally through some process.
2. Use this file through S3 CDK deployment to be uploaded to an arbitary bucket. (This file is then supposed to be used to deserialize protobuf messages elsewhere)

I managed to upload the file. But when I downloaded the file and performed a diff, it seems that the file has been manipulated/tampered with. In particular some encoded non utf-8 characters seem off. The differences are provided in the Current behavior section

It might be that this code here : https://github.com/aws/aws-cdk/blob/f7a65ea668d6b2ae8be192ab8f93b7a83b94e87e/packages/%40aws-cdk/aws-s3-deployment/lib/lambda/index.py#L292 is deliberately replacing lines here and hence the file is incorrect. The file is effectively unusable since the protobuf wants an exact serialized replica of the file.

We tried to use the https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3_assets-readme.html and it seems to have worked but this aws s3 bucket deploy does not.

### Expected Behavior

The local and uploaded file should be exactly the same.

### Current Behavior

If we upload the file through CDK BucketDeployment and download it there are certain characters that are missing.

Some example diff being

```
➜ diff main.desc uploadedmain.desc
2c2
<
---
> �
7c7
<
---
> �
13c13
<
---
> �
20c20
<
---
```

### Reproduction Steps

1. Create a https://googleapis.dev/python/protobuf/3.11.2/google/protobuf/descriptor.html file. We can provide a sample one for to test but the problem occurs in arbitrary file descriptor sets.

2. Create a bucket deployment asset on the directory that contains the fd set. Perform a stack deployment to upload.

3. Use S3 cli to download the newly uploaded artifact

### Possible Solution

I do not really have one. I suspect its happening here. https://github.com/aws/aws-cdk/blob/f7a65ea668d6b2ae8be192ab8f93b7a83b94e87e/packages/%40aws-cdk/aws-s3-deployment/lib/lambda/index.py#L298 but then again I do not know of the implications of removing/editing that code and if it would produce breaking changes for those who use this for simple text files.

### Additional Information/Context

_No response_

### CDK CLI Version

2.40.0

### Framework Version

_No response_

### Node.js Version

14

### OS

Mac OS X

### Language

Typescript

### Language Version

4.1.2

### Other information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with packages/@aws-cdk/aws-s3-deployment/lib/lambda/index.py at the referenced lines and reproduce the BucketDeployment upload using a protobuf descriptor set. Compare the local file with the S3-downloaded file byte-for-byte; done means binary files remain identical while the stated text-file behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.