aws / aws/containers-roadmap

[ECS/CodeDeploy] [request]: Support uncompressed streaming zip

Open
#1,112 0 comments 1 reaction 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Please fix CodePipeline's _Amazon ECS (Blue/Green)_ action so that it can unzip files that are produced by streaming _and_ use STORE as the archive method. These files cannot be extracted by Java's standard java.util.zip, which appears to be in use by the action.

**Which service(s) is this request for?**
AWS CodePipeline's _Amazon ECS (Blue/Green)_ action, affecting CodeDeploy and ECS.

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
The CodePipeline action _Amazon ECS (Blue/Green)_ does not support zip files that are created by streaming _and_ use STORE (not DEFLATE) as the archive method. The error produced by the action is:

> Exception while trying to read the task definition artifact file from: \

The zip file is produced by Terraform's `archive_file` provider, but it has been patched with https://github.com/saveriomiroddi/terraform-provider-archive-dev/commit/347a1af40e706e32287b2e9bc32f9ff91870ed31 to ensure that file metadata is normalized, i.e. that they are the same across platforms. To do this, it sets mtime and permissions, but also forces the STORE archive method to avoid minute differences in compression output across platforms.

See https://github.com/hashicorp/terraform-provider-archive/pull/47 for details on the patch.

The combination of Go's streaming zip writing (used by `archive_file` under the hood) and the STORE archive method, seems to be the the core of the problem.

Speculation: The AWS CodePipeline _Amazon ECS (Blue/Green)_ action may be using the standard java.util.zip package to extract the archive containing `appspec.yaml` and `taskdef.json` for CodeDeploy and ECS respectively.

A test program using java.util.zip reproduces the problem:
```
java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
at java.base/java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:311)
at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:123)
at com.meh.test.UnzipFiles.unzip(UnzipFiles.java:30)
at com.meh.test.UnzipFiles.main(UnzipFiles.java:17)
```

Relevant links:
* https://bugs.openjdk.java.net/browse/JDK-8143613
* https://groups.google.com/g/golang-nuts/c/0iae5Ng-I-0
* https://github.com/30x/zipper
* https://stackoverflow.com/questions/47208272/android-zipinputstream-only-deflated-entries-can-have-ext-descriptor

**Are you currently working around this issue?**
We have reverted to not use the patched `archive_file` Terraform provider, and rather dealing with file permission problems manually, which sucks. The main challenge is differences in default umasks between OSX, Ubuntu and other Linux distributions.

I have reported the issue upstream to the patched Terraform provider too: https://github.com/saveriomiroddi/terraform-provider-archive-dev/issues/1

**Attachments**
The attached `test.zip` has been created by the patched `archive_file` Terraform provider. It reproduces the CodePipeline exception, and the java.util.zip exception.

[test.zip](https://github.com/aws/containers-roadmap/files/5367249/test.zip)
[UnzipFiles.zip](https://github.com/aws/containers-roadmap/files/5372175/UnzipFiles.zip)

```
$ javac UnzipFiles.java
$ java -cp . UnzipFiles
java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
at java.base/java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:311)
at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:123)
at UnzipFiles.unzip(UnzipFiles.java:23)
at UnzipFiles.main(UnzipFiles.java:12)

$ java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode)
```

Please note: While the zip only contains a dummy file `test1.txt` with body `hello`, it produces exactly the same error as a perfectly valid pair of `appspec.yaml` and `taskdef.json` files.

Contributor guide

Open the contributing guide

Research direction

Start with the attached test.zip and UnzipFiles.zip, then reproduce the failure using the Java java.util.zip example described in the issue. Trace the AWS CodePipeline Amazon ECS (Blue/Green) action's handling of the artifact; done means archives written as streaming STORE entries can be extracted for appspec.yaml and taskdef.json without the reported exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, terraform
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.