apache / apache/incubator-xtable

spark-runtime: no way to configure Parquet source partitioning; targets always come out unpartitioned

Open
#927 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

### Describe the bug

The 0.4.0 release notes list the Parquet source "with partition extraction", and the core module implements it (`ParquetSourceConfig` / `ParquetPartitionSpecExtractor`, exercised directly by the core integration tests). However, the spark-runtime bundle, a shipped way to run 0.4.0 syncs, provides **no route to that configuration**: converting a Hive-style partitioned Parquet table through `XTableSparkSync` or `XTableSyncService` always produces an unpartitioned target.

`--partitionspec` does not help, and per the CLI help and `TableSyncSpec` javadoc it is scoped to Hudi sources, so this is a missing-capability gap in the runtime rather than a broken flag. A secondary UX point: the CLI accepts `--partitionspec` together with `--sourceformat PARQUET` without any warning, which reads as if it applies; failing fast (or forwarding it) would remove the ambiguity.

### Root cause

`XTableSyncService` forwards `TableSyncSpec.partitionSpec` using only the Hudi configuration key, for every source format: https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/xtable-spark-runtime/src/main/java/org/apache/xtable/spark/XTableSyncService.java#L62-L65

```java
if (spec.getPartitionSpec() != null && !spec.getPartitionSpec().isEmpty()) {
sourceProperties.put(HudiSourceConfig.PARTITION_FIELD_SPEC_CONFIG, spec.getPartitionSpec());
}
```

The Parquet source reads a different key, `xtable.parquet.source.partition_field_spec_config`, which nothing in the spark-runtime ever populates: https://github.com/apache/incubator-xtable/blob/e8f22867a6cb297f58b248150220fce71188d16a/xtable-core/src/main/java/org/apache/xtable/parquet/ParquetSourceConfig.java#L36-L74

Related but distinct: #901 covers the schema being derived from a single file footer, which independently drops directory-only partition columns for Spark-written (`partitionBy`) layouts. Wiring the configuration through is necessary but may not be sufficient for such layouts until #901 is addressed.

### To Reproduce

To isolate this from #901 (footer-derived schema), the fixture keeps `order_date` **inside the Parquet files** while still using Hive-style partition directories: files are written directly into `order_date=2026-08-01/` without Spark's `partitionBy` (which would strip the column from the files). Reproduction scripts are included at the bottom of this issue.

```bash
# 1. Hive-style layout, partition column RETAINED in the file schema
spark-submit gen_parquet_keepcol.py /tmp/orders_parquet_kc

# 2. sync, with and without the flag (same result)
spark-submit --master 'local[2]' \
--class org.apache.xtable.spark.XTableSparkSync \
--jars iceberg-spark-runtime-3.5_2.12-1.9.2.jar \
xtable-spark-runtime_2.12-0.4.0-incubating.jar \
--basepath /tmp/orders_parquet_kc --sourceformat PARQUET --targets ICEBERG \
--partitionspec order_date:VALUE

# 3. inspect
spark-submit --jars iceberg-spark-runtime-3.5_2.12-1.9.2.jar \
inspect_iceberg.py /tmp/orders_parquet_kc --partitions
```

Observed: `order_date` **is present** in the target schema (so #901 is not in play for this fixture), yet the table has an empty partition spec and no partition values, identically with and without `--partitionspec`. For contrast, a Hudi source with the same flag on the same runtime produces a correctly partitioned target, confirming the flag transport works when the matching config key is written.

### Expected behavior

The spark-runtime provides a way to configure Parquet source partitioning (e.g. forwarding the spec to the source-format-appropriate config key), so that the release-notes capability is reachable through the shipped runtime; and `--partitionspec` with a PARQUET source either applies or fails fast with a clear message.

### Affected versions

`0.4.0-incubating` artifact; code verified identical on `branch-0.4` HEAD (the spark-runtime module does not exist on `main` at the time of filing).

#### Reproduction scripts

gen_parquet_keepcol.py

```python
import sys
from pyspark.sql import SparkSession, functions as F
base = sys.argv[1]
spark = SparkSession.builder.getOrCreate()
for d, off in [("2026-08-01", 0), ("2026-08-02", 1000)]:
(spark.range(off, off + 1000).withColumn("order_id", F.col("id"))
.withColumn("order_date", F.lit(d)).drop("id").repartition(2)
.write.mode("overwrite").parquet(f"{base}/order_date={d}")) # no partitionBy
print("wrote 2000 rows across 2 partition dirs ->", base)
```

inspect_iceberg.py

```python
import sys
from pyspark.sql import SparkSession, functions as F
path = sys.argv[1]
spark = SparkSession.builder.getOrCreate()
df = spark.read.format("iceberg").load(path)
print("optimized count(*) :", df.count())
print("forced distinct order_id:", df.select(F.countDistinct("order_id")).collect()[0][0])
print("columns :", df.columns)
spark.read.format("iceberg").load(path + "#files") \
.select("file_path", "record_count", "value_counts").show(truncate=False)
if "--partitions" in sys.argv:
try:
spark.read.format("iceberg").load(path + "#partitions").show(truncate=False)
except Exception as e:
print("partitions metadata unavailable:", type(e).__name__)
```

### Are you willing to submit PR?

- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in xtable-spark-runtime/src/main/java/org/apache/xtable/spark/XTableSyncService.java and follow XTableSparkSync/TableSyncSpec into the source configuration. Compare its forwarding with xtable-core/src/main/java/org/apache/xtable/parquet/ParquetSourceConfig.java, then run the included reproduction scripts. Done means Parquet partition configuration reaches the runtime and the CLI either applies it or clearly rejects the incompatible flag.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.