apache / apache/polaris

Make all generated archives reproducible

Open
#2,204 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

To be able to validate a release artifact (jar, tarball, zip, etc) against the state of the source tree, the artifacts need to be generated in a reproducible way.

There are a few, totally build related differences as of today:
1. [x] jar files: `META-INF/MANIFEST.MF` have different attributes (**fixed**)
* `Apache-Polaris-Is-Release` (depends on the `-Prelease` vs just `-PjarWithGitinfo` property)
* `Apache-Polaris-Build-Git-Describe` (same as above)
* `Apache-Polaris-Build-Timestamp` - could be changed to the Git commit timestamp using UTC
* `Apache-Polaris-Build-System` - remove this one
* `Apache-Polaris-Build-Java-Version` - remove this one
2. [x] jar files: `META-INF/jandex.idx` are reported to be different (**fixed**)
3. [x] jar files: mtime of `META-INF/MANIFEST.MF` and `META-INF/jandex.idx` vary (**fixed**)
4. [ ] `..../quarkus/generated-bytecode.jar` class files differ. Since Quarkus 3.28.2, the generated bytecode is often the same, but there is still no guarantee.
5. [x] varying order of entries in Quarkus re-assembled `.../app/polaris-*.jar` archives (https://github.com/quarkusio/quarkus/issues/50578)
6. [x] "group write" POSIX permission varies between platforms for zip/tar archive entries (fixed via #2819 + #3146)
With all mentioned issues addressed, it should be possible to build distribution artifacts that match binary, leading to the same sha512.
7. [x] source tarballs have entries with "current mtime". The source tarballs should use a fixed mtime (fixed via #2823)
8. [x] "Top level POM" is not reproducible, because the POM `` and `` elements are dynamically populated at build time (fixed via #2826).
9. [ ] Helm chart package tarball (generated via `helm package`) is not reproducible, `helm package` has no options to "fix" entries' mtime and POSIX attributes. See: Releasy: make Helm package reproducible (#3086)
10. [x] Source-tarball - eliminate git-gzip risk (via #3075)
11. [ ] Build: Ensure reproducible .properties files (via #3089)
12. [X] `git archive --mtime` requires a timezone (via #3143)
13. [x] Non deterministic behavior with `withXml()` for `pom.xml` customization (via #3145)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the three remaining items: Quarkus generated-bytecode.jar, Helm package tarballs, and generated .properties files; read the linked #3086 and #3089 issues for the Helm and properties work. Done means the remaining generated artifacts are reproducible so distribution builds match binary and produce the same sha512.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, helm, java
Domain
build-system, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.