Audit third-party code bundled in published JARs
- 主要言語
- Java
- スター
- 3.1k
- フォーク
- 1.6k
- 平均マージ
- 3日 12時間
- マージ済み PR(30日)
- 33
説明
## Background
We need to identify which published JARs bundle third-party code. ASF release policy requires JARs to place [LICENSE and NOTICE in `META-INF`](https://www.apache.org/legal/release-policy.html#what-are-the-requirements-to-distribute-other-artifacts-in-addition-to-the-source-package) and requires the [LICENSE file to cover every bundled non-Apache component](https://www.apache.org/legal/release-policy.html#the-license-file). Any [required third-party notices](https://www.apache.org/legal/resolved.html#what-are-required-third-party-notices) also need to be included.
This issue documents the findings and provides a script to verify them.
## Findings
Cross-checking the final JAR contents against the Maven Shade configuration found:
- **7 binary JARs with bundled third-party classes** (**these need to be reflected in their respective LICENSE and NOTICE files**)
- 13 Javadoc JARs with JDK-generated assets
- 39 JARs with no bundled third-party code
| Artifact | Maven configuration | Final JAR contents |
|---|---|---|
| `parquet-avro-*.jar` | Explicitly shades FastUtil | FastUtil |
| `parquet-column-*.jar` | Inherits FastUtil/OpenHFT Shade rules | FastUtil, OpenHFT |
| `parquet-format-structures-*.jar` | Explicitly includes `libthrift` | Apache Thrift |
| `parquet-hadoop-*.jar` | Shade processes Jackson and FastUtil; filters/minimization remove Jackson | FastUtil only |
| `parquet-jackson-*.jar` | Explicitly includes Jackson artifacts | Jackson |
| `parquet-hadoop-bundle-*.jar` | Includes other `org.apache.parquet:parquet-*` JARs | Carries their already-shaded third-party classes |
| `parquet-cli-*-runtime.jar` | `shadedArtifactAttached=true`, classifier `runtime`, includes all dependencies | Full runtime dependency set |
A few edge cases:
- `parquet-variant`, `parquet-protobuf`, and `parquet-thrift` run Shade, but filtering and minimization leave no third-party classes in the final JARs.
- `parquet-benchmarks.jar` bundles the full dependency tree, but it is a local-only uber-JAR. The published versioned `parquet-benchmarks-*.jar` contains only project classes.
- The 15 source and 15 test JARs contain project output rather than dependency code.
- The 13 Javadoc JARs contain JDK-generated JavaScript and CSS. [ASF policy explicitly permits these assets in Maven Javadoc JARs](https://www.apache.org/legal/resolved.html#from-java-9-onwards-javadoc-can-include-search-functionality-that-includes-javascript-under-other-open-source-licenses-can-apache-projects-include-this-javadoc).
Setup and verification commands
### Setup
Run the setup commands from the repository root with Java 17. Maven output is left visible so build progress and failures are easy to follow.
```bash
java -version
./mvnw clean package --batch-mode -DskipTests
./mvnw --batch-mode process-resources source:jar-no-fork javadoc:jar
```
### Verification
After setup completes, run the verification separately:
```bash
bash <<'EOF'
set -euo pipefail
export LC_ALL=C
tmp_dir=$(mktemp -d)
trap 'rm -rf "$tmp_dir"' EXIT
candidate_jars="$tmp_dir/candidates"
third_party_jars="$tmp_dir/third-party"
javadoc_jars="$tmp_dir/javadocs"
clean_jars="$tmp_dir/clean"
: >"$third_party_jars"
: >"$javadoc_jars"
: >"$clean_jars"
find . -mindepth 3 -maxdepth 3 -type f \
-path './*/target/*.jar' \
! -name 'original-*.jar' \
! -name 'parquet-benchmarks.jar' \
-print0 >"$candidate_jars"
count_non_parquet_classes() {
jar tf "$1" | awk '
/\.class$/ {
path=$0
sub(/^META-INF\/versions\/[0-9]+\//, "", path)
if (path !~ /^org\/apache\/parquet\//) count++
}
END { print count+0 }
'
}
count_external_sources() {
jar tf "$1" | awk '
/\.java$/ && $0 !~ /^org\/apache\/parquet\// { count++ }
END { print count+0 }
'
}
count_extra_test_classes() {
local jar_file=$1
local module_dir=$2
local jar_classes="$tmp_dir/jar-classes"
local project_classes="$tmp_dir/project-classes"
jar tf "$jar_file" | awk '/\.class$/' | sort -u >"$jar_classes"
: >"$project_classes"
if [[ -d "$module_dir/target/test-classes" ]]; then
(
cd "$module_dir/target/test-classes"
find . -type f -name '*.class' -print |
sed 's#^\./##' |
sort -u
) >"$project_classes"
fi
comm -13 "$project_classes" "$jar_classes" | awk 'END { print NR+0 }'
}
printf 'Inspecting existing JARs...\n'
jar_count=0
while IFS= read -r -d '' jar_file; do
jar_count=$((jar_count + 1))
jar_name=$(basename "$jar_file")
module_dir=${jar_file%/target/*}
case "$jar_name" in
*-javadoc.jar)
unzip -p "$jar_file" script.js >"$tmp_dir/script.js"
if grep -Fq 'Oracle and/or its affiliates' "$tmp_dir/script.js"; then
printf '%s\tJDK-generated JavaScript/CSS\n' "$jar_file" >>"$javadoc_jars"
else
printf '%s\n' "$jar_file" >>"$clean_jars"
fi
continue
;;
*-sources.jar)
bundled_count=$(count_external_sources "$jar_file")
;;
*-tests.jar)
bundled_count=$(count_extra_test_classes "$jar_file" "$module_dir")
;;
*)
bundled_count=$(count_non_parquet_classes "$jar_file")
;;
esac
if [[ $bundled_count -gt 0 ]]; then
printf '%s\t%d bundled classes or sources\n' \
"$jar_file" "$bundled_count" >>"$third_party_jars"
else
printf '%s\n' "$jar_file" >>"$clean_jars"
fi
done <"$candidate_jars"
if [[ $jar_count -eq 0 ]]; then
printf 'No JARs found. Run the setup commands first.\n' >&2
exit 1
fi
printf '\n=== JARs containing bundled third-party code ===\n\n'
sort "$third_party_jars"
printf '\n=== Javadoc JARs containing JDK-provided assets ===\n\n'
sort "$javadoc_jars"
printf '\n=== JARs with no bundled third-party code found ===\n\n'
sort "$clean_jars"
printf '\nInspected %d published JARs.\n' "$jar_count"
EOF
```
The Java 17 build completed successfully, and the legal-file check verified all 59 JARs.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Java 17 を使用してリポジトリのルートから開始し、続いて記載されている Maven セットアップコマンドを実行して、公開されている JAR とサポート用アーティファクトをビルドします。提供されている Bash 検証スクリプトは個別に実行し、同梱されたクラス、ソース、テスト、Javadoc アセットを調査します。調査結果が文書化され、検証が 59 個の JAR を対象とし、legal-file check に合格すれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- bash, java
- 領域
- build-system, documentation, release
- issue の種類
- ドキュメント
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 静か
- 明瞭さ
- 説明が足りない
- 初心者へのやさしさ
- 48/100