(aws-cdk-deployment): (AWS CDK deployment modfies binary data files that are non utf-8 encoded before uploading them to S3)
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
I was trying to use https://docs.aws.amazon.com/cdk/api/v1/docs/aws-s3-deployment-readme.html to upload a binary file. My specific use case is to
1. Create a google protobuf file descriptor set file. https://googleapis.dev/python/protobuf/3.11.2/google/protobuf/descriptor.html locally through some process.
2. Use this file through S3 CDK deployment to be uploaded to an arbitary bucket. (This file is then supposed to be used to deserialize protobuf messages elsewhere)
I managed to upload the file. But when I downloaded the file and performed a diff, it seems that the file has been manipulated/tampered with. In particular some encoded non utf-8 characters seem off. The differences are provided in the Current behavior section
It might be that this code here : https://github.com/aws/aws-cdk/blob/f7a65ea668d6b2ae8be192ab8f93b7a83b94e87e/packages/%40aws-cdk/aws-s3-deployment/lib/lambda/index.py#L292 is deliberately replacing lines here and hence the file is incorrect. The file is effectively unusable since the protobuf wants an exact serialized replica of the file.
We tried to use the https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_s3_assets-readme.html and it seems to have worked but this aws s3 bucket deploy does not.
### Expected Behavior
The local and uploaded file should be exactly the same.
### Current Behavior
If we upload the file through CDK BucketDeployment and download it there are certain characters that are missing.
Some example diff being
```
➜ diff main.desc uploadedmain.desc
2c2
<
---
> �
7c7
<
---
> �
13c13
<
---
> �
20c20
<
---
```
### Reproduction Steps
1. Create a https://googleapis.dev/python/protobuf/3.11.2/google/protobuf/descriptor.html file. We can provide a sample one for to test but the problem occurs in arbitrary file descriptor sets.
2. Create a bucket deployment asset on the directory that contains the fd set. Perform a stack deployment to upload.
3. Use S3 cli to download the newly uploaded artifact
### Possible Solution
I do not really have one. I suspect its happening here. https://github.com/aws/aws-cdk/blob/f7a65ea668d6b2ae8be192ab8f93b7a83b94e87e/packages/%40aws-cdk/aws-s3-deployment/lib/lambda/index.py#L298 but then again I do not know of the implications of removing/editing that code and if it would produce breaking changes for those who use this for simple text files.
### Additional Information/Context
_No response_
### CDK CLI Version
2.40.0
### Framework Version
_No response_
### Node.js Version
14
### OS
Mac OS X
### Language
Typescript
### Language Version
4.1.2
### Other information
_No response_
Contributor guide
Research direction
Start with packages/@aws-cdk/aws-s3-deployment/lib/lambda/index.py at the referenced lines and reproduce the BucketDeployment upload using a protobuf descriptor set. Compare the local file with the S3-downloaded file byte-for-byte; done means binary files remain identical while the stated text-file behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100