Spark: Remove duplicated TestTables and use core test helper
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
## Problem
Spark v3.5, v4.0, and v4.1 each have a duplicate `TestTables` helper (`spark/*/spark/src/test/java/org/apache/iceberg/spark/source/TestTables.java:42`) with a TODO to use the core version instead.
Core already has a richer `org.apache.iceberg.TestTables` in `core/src/test/java/org/apache/iceberg/TestTables.java`. The Spark copies are outdated and risk drifting from core test behavior.
Affected Spark tests: `TestBaseReader`, `TestSparkReadProjection`, `TestSparkMetadataColumns`, `TestSparkCatalog`.
## Proposed behavior
1. Delete the three Spark `TestTables` copies.
2. Switch affected tests to `org.apache.iceberg.TestTables` (already available via `iceberg-core` test artifacts).
3. Update call sites for API differences:
- `create(...)` requires a `formatVersion` argument
- `load(name)` → `load(File temp, String name)`
- `replaceMetadata(...)` is not in core — add it there or refactor `TestSparkReadProjection` to commit via `TableOperations`
## Scope
- Test-only refactor across `spark/v3.5`, `spark/v4.0`, `spark/v4.1`
- ~4 test classes per Spark version
- May need small additions to core `TestTables` (e.g. `replaceMetadata`, public `readMetadata`)
- `TestSparkCatalog` needs extra care: core keys tables by `(location, name)`, not name alone
## Notes
- No production code changes or new dependencies.
- Can be done one Spark version at a time or all three together.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start by comparing the three Spark TestTables.java copies with core/src/test/java/org/apache/iceberg/TestTables.java, then inspect TestBaseReader, TestSparkReadProjection, TestSparkMetadataColumns, and TestSparkCatalog for API differences. Run the affected Spark tests for each version. Done means the duplicate helpers are removed, call sites use the core helper, and all affected tests pass without production changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100