apache / apache/iceberg

[Spark Integration Tests] TestCreateTable::testCreateTableCommitProperties won't work on RESTCatalog

Open
#11,554 4 comments 0 reactions 0 assignees View on GitHub
bug not-stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

None

### Query engine

None

### Please describe the bug 🐞

Part of: https://github.com/apache/iceberg/issues/11079

## Intro
The test `TestCreateTable::testCreateTableCommitProperties` will only work on Hadoop/Hive Catalog but not on REST Catalog. Code link: https://github.com/apache/iceberg/blame/0a705b0637db484730eb4eece69ae6c4d52fd9da/spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java#L353

## Error message
```
TestCreateTable > testCreateTableCommitProperties() > catalogName = testrest, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=rest, cache-enabled=false, uri=http://localhost:34885/} FAILED
java.lang.AssertionError:
Expecting actual throwable to be an instance of:
org.apache.iceberg.exceptions.ValidationException
but was:
org.apache.iceberg.exceptions.BadRequestException: Malformed request: Table property commit.retry.num-retries must have integer value
at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:208)
at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:118)
at org.apache.iceberg.rest.ErrorHandlers$TableErrorHandler.accept(ErrorHandlers.java:102)
...(164 remaining lines not displayed - this can be changed with Assertions.setMaxStackTraceElementsDisplayed)
at org.apache.iceberg.spark.sql.TestCreateTable.testCreateTableCommitProperties(TestCreateTable.java:366)
```

## Root cause
REST client will throw `BadRequestException` (`HTTP 400`) where a `ValidationException` will be thrown by Hive/Hadoop Catalog in similar situation. REST client will simply categorize `ValidationException` as `HTTP 400` and throw `BadRequestException` instead. This is currently determined by REST's `ErrorHandlers`: https://github.com/apache/iceberg/blob/0a705b0637db484730eb4eece69ae6c4d52fd9da/core/src/main/java/org/apache/iceberg/rest/ErrorHandlers.java#L202

## Further consideration
Two options going forward:
1. Unify Hive/Hadoop/REST interface so all of them throw the same type of exception in the cases that this test is covering for
2. Accept that Hive/Hadoop/REST could throw different type of exception in these cases and modify the test itself

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.