apache / apache/iceberg

ExpireSnapshotsSparkAction fails with NullPointerException when TableMetadata.metadataFileLocation() is null

Open Beginner friendly
#16,838 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.9.2

### Query engine

Spark

### Please describe the bug 🐞

Iceberg Version

* Apache Iceberg: 1.9.2
* Spark Runtime: iceberg-spark-runtime-3.5
* Spark Version: 3.5.7

Description

While implementing a custom Iceberg REST Catalog for our product, I encountered a NullPointerException during snapshot expiration.

The root cause turned out to be that my REST Catalog accidentally returned:

{
"metadata_location": "..."
}

instead of the expected REST field:

{
"metadata-location": "..."
}

Because of this typo, TableMetadata.metadataFileLocation() becomes null.

Interestingly, normal table reads continue to work correctly because Iceberg is still able to discover and load the current metadata through the table location.

However, after a successful remove-snapshots commit, ExpireSnapshotsSparkAction reloads the table metadata from the catalog and later constructs a static table using:

protected Table newStaticTable(TableMetadata metadata, FileIO io) {
StaticTableOperations ops = new StaticTableOperations(metadata, io);
return new BaseTable(ops, metadata.metadataFileLocation());
}

Since metadata.metadataFileLocation() is null, the resulting BaseTable is created with a null metadata location.

Later, during file expiration (fileDS()), this causes an unexpected NullPointerException inside the Spark job execution.

Relevant code path

private Dataset fileDS(TableMetadata metadata, Set snapshotIds) {
Table staticTable = this.newStaticTable(metadata, this.table.io());
return this.contentFileDS(staticTable, snapshotIds)
.union(this.manifestDS(staticTable, snapshotIds))
.union(this.manifestListDS(staticTable, snapshotIds))
.union(this.statisticsFileDS(staticTable, snapshotIds));
}

which eventually calls:

protected Table newStaticTable(TableMetadata metadata, FileIO io) {
StaticTableOperations ops = new StaticTableOperations(metadata, io);
return new BaseTable(ops, metadata.metadataFileLocation());
}

Expected behavior

If metadata.metadataFileLocation() is required for ExpireSnapshotsSparkAction, Iceberg should fail fast with a descriptive error such as:

Table metadata file location is null.
This may indicate an invalid REST Catalog response or missing metadata-location field.

instead of continuing and eventually failing with an unrelated NullPointerException.

Actual behavior

The action proceeds until later stages and eventually fails with a NullPointerException, making the root cause difficult to identify.

Root cause

In my case, the issue was an incorrect REST Catalog response field name (metadata_location instead of metadata-location).

After fixing the REST response to return the correct metadata-location property, the issue disappeared completely.

Suggestion

It would be helpful to add an explicit null check around metadata.metadataFileLocation() in BaseSparkAction.newStaticTable() (or earlier) and throw a meaningful exception explaining that the REST Catalog returned an invalid or incomplete table metadata response.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start at BaseSparkAction.newStaticTable(), the entry point named in the issue, and trace how its metadata is used by ExpireSnapshotsSparkAction.fileDS(). Check for existing tests covering this path; done means a null metadata file location produces a descriptive error rather than a later NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.