apache / apache/iceberg

Spark: Support custom hash distribution columns for unpartitioned tables with graceful fallback

Open
#16,767 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

### Problem Description

When write.distribution-mode=hash is set on an unpartitioned Iceberg table, the system silently downgrades the distribution mode to NONE because there are no partition columns to hash-distribute by. This happens in SparkWriteConf.adjustWriteDistributionMode():
} else if (mode == HASH && table.spec().isUnpartitioned()) {
return NONE;
}
This means users cannot control the number of output files for unpartitioned tables via hash distribution. They are forced into NONE mode where each Spark task writes its own file, leading to either too many small files (fanout) or too few large files depending on upstream parallelism.

### Current Behavior

1. Set write.distribution-mode=hash on an unpartitioned table → distribution mode is silently downgraded to NONE
2. No mechanism exists to specify which columns to hash-distribute by for unpartitioned tables
3. If a user manually configures distribution columns that don't exist in the table schema, the write fails at runtime with an opaque Spark error (NamedReference to non-existent column)

### Expected Behavior

1. Users can specify custom columns for hash distribution on unpartitioned tables via a new distribution-columns configuration
2. When configured, HASH mode is preserved (not downgraded) and data is shuffled by the specified columns
3. A "*" wildcard expands to all schema columns dynamically
4. An ignore-missing fallback option gracefully handles columns that don't exist in the schema by falling back to sort-order columns or all columns

### Motivation

This feature was motivated by a production use-case where users writing to unpartitioned streaming tables needed to control output file counts to avoid overwhelming downstream readers. Without hash distribution on unpartitioned tables, file counts are dictated by upstream task parallelism and cannot be tuned independently.

The ignore-missing fallback was added to handle cases where distribution-columns are configured at a catalog level but referenced columns may not exist in all target tables, preventing runtime failures while maintaining meaningful distribution where possible.

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start in SparkWriteConf.adjustWriteDistributionMode(), then trace the existing write distribution configuration and validation paths. Implement custom distribution columns for unpartitioned tables, including wildcard expansion and the ignore-missing fallback described in the issue. Done means HASH is preserved for valid configured columns, missing columns no longer cause an opaque runtime failure, and fallback distribution uses sort-order or schema columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.