apache / apache/incubator-xtable
Shrink the xtable-utilities bundled jar
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
### Feature Request / Improvement
`xtable-utilities` is 795 lines of Java in two classes, `RunSync` and `RunCatalogSync`. It builds a 1.1 GB bundled jar: 3.1 GB uncompressed, 582,980 entries, 483 whitelist entries.
Measured by top-level package:
| | MB uncompressed | files |
|---|---:|---:|
| `software/amazon/awssdk` | 1957 | 379,410 |
| `org/apache/hadoop` | 258 | 53,612 |
| `org/apache/spark` | 91 | 12,129 |
| `org/apache/paimon` | 86 | 18,840 |
| RocksDB natives | 159 | 13 |
| `io/grpc/xds` | 40 | 4,805 |
| `it/unimi/dsi` | 39 | 10,665 |
| `org/apache/hudi` | 17 | 2,890 |
| `org/apache/iceberg` | 12 | 3,126 |
The formats XTable exists to translate are about 1% of the jar.
This is the "reduce the size of bundled jar" step from the 2024 [DISCUSS] thread on excluding `xtable-utilities` from the release, where @vinothchandar suggested moving to the Hudi model of whitelisting what compute engines do not already provide. The whitelist has grown rather than shrunk since.
### What can be cut
Four changes, no source touched, all measured:
1. `software.amazon.awssdk:bundle` is the all-services jar. `hadoop-aws` 3.4.1's S3A references exactly one service package, `services/s3`, plus `STSClient`, and the `transfer`, `http`, `core`, `awscore` and `auth` packages. Nothing else. Verified by scanning the constant pools of all 418 `org.apache.hadoop.fs.s3a` classes. Replacing `bundle` with `s3`, `sts`, `netty-nio-client` and `s3-transfer-manager` takes the SDK from 1957 MB to 50 MB.
2. RocksDB ships a native library per platform. Dropping ppc64le, s390x and 32-bit Linux removes 84 MB. x86_64 and aarch64 keep their musl builds; macOS and Windows stay for development.
3. `io.grpc:grpc-xds` is 40 MB, referenced only from gRPC's own code, and reachable only through `xds://` targets.
4. `org.codehaus.groovy:groovy-all` 2.4.11 arrives via Hive; nothing outside Groovy references it.
Together: **1098 MB to 423 MB**, entries 582,980 to 203,041.
### What cannot be cut without a design change
Spark's 91 MB is load-bearing today: `xtable-core`'s Delta support uses `delta-core`, a Spark library, across 12 classes (`SparkSession`, `DeltaLog`, `StructType`). `delta-kernel-api` and `delta-kernel-defaults` are already dependencies at 4.0.0. Kernel is the Spark-free Delta API, so migrating that code would let Spark and much of its transitive weight go. That is a larger piece of work and relates to #611.
`hadoop-azure` and the GCS connector have no static referrer, but they are `ServiceLoader`-loaded filesystems, so their absence would be a lost feature rather than dead weight.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
*This issue was created with AI assistance.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Locate the xtable-utilities dependency declarations and bundled-jar packaging configuration; RunSync and RunCatalogSync are the only source classes named. Verify the four dependency exclusions and replacements without changing source, then rebuild and confirm the jar is about 423 MB with roughly 203,041 entries while retaining the stated platform binaries and filesystem features.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, groovy, java, spark
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100