apache / apache/iceberg

Spark: Remove duplicated TestTables and use core test helper

Open
#17,508 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

## Problem

Spark v3.5, v4.0, and v4.1 each have a duplicate `TestTables` helper (`spark/*/spark/src/test/java/org/apache/iceberg/spark/source/TestTables.java:42`) with a TODO to use the core version instead.

Core already has a richer `org.apache.iceberg.TestTables` in `core/src/test/java/org/apache/iceberg/TestTables.java`. The Spark copies are outdated and risk drifting from core test behavior.

Affected Spark tests: `TestBaseReader`, `TestSparkReadProjection`, `TestSparkMetadataColumns`, `TestSparkCatalog`.

## Proposed behavior

1. Delete the three Spark `TestTables` copies.
2. Switch affected tests to `org.apache.iceberg.TestTables` (already available via `iceberg-core` test artifacts).
3. Update call sites for API differences:
- `create(...)` requires a `formatVersion` argument
- `load(name)` → `load(File temp, String name)`
- `replaceMetadata(...)` is not in core — add it there or refactor `TestSparkReadProjection` to commit via `TableOperations`

## Scope

- Test-only refactor across `spark/v3.5`, `spark/v4.0`, `spark/v4.1`
- ~4 test classes per Spark version
- May need small additions to core `TestTables` (e.g. `replaceMetadata`, public `readMetadata`)
- `TestSparkCatalog` needs extra care: core keys tables by `(location, name)`, not name alone

## Notes

- No production code changes or new dependencies.
- Can be done one Spark version at a time or all three together.

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by comparing the three Spark TestTables.java copies with core/src/test/java/org/apache/iceberg/TestTables.java, then inspect TestBaseReader, TestSparkReadProjection, TestSparkMetadataColumns, and TestSparkCatalog for API differences. Run the affected Spark tests for each version. Done means the duplicate helpers are removed, call sites use the core helper, and all affected tests pass without production changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.