[Bug] Iceberg REST retries Aliyun OSS NoSuchKey and holds Gravitino tree locks for missing metadata files
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What happened
When an Iceberg table registered in Gravitino points to an Aliyun OSS metadata JSON that no longer exists, Gravitino's Iceberg REST service can spend a long time retrying the missing object. During that time, Gravitino table/tag/policy requests may hold tree locks for more than 30 seconds, causing the UI and metadata APIs for the same schema to appear stuck.
### Why this matters
A single stale or externally damaged Iceberg table can degrade a whole schema's metadata browsing experience. This is especially visible in the web UI, which may issue table detail, tags, policies, and existence checks close together.
### Observed behavior
For a table similar to `metalake.catalog.schema.table`, where the JDBC Iceberg catalog row still pointed at a missing metadata file, the Gravitino server log repeatedly showed:
```text
Retrying task after failure: sleepTimeMs=... The specified key does not exist.
.../metadata/00000-...metadata.json
```
The same time window also showed tree-lock watchdog warnings similar to:
```text
Thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-...,5,main], ident=metalake.catalog.schema.table} holds the lock node TreeLockNode{ident=/,...} for more than 30s ... io-connection hangs
```
After unregistering/dropping the stale table entry, the same API returned quickly with `NoSuchTableException`/404.
### Expected behavior
Missing Aliyun OSS metadata objects should be treated as a non-retryable not-found condition and should fail quickly. A stale table should not cause long retry loops that hold Gravitino tree locks and block unrelated metadata browsing in the same schema.
### Root cause analysis
Gravitino 1.2.0 uses Iceberg 1.10.1. Iceberg's `BaseMetastoreTableOperations.refreshFromMetadataLocation` stops retrying on Iceberg `NotFoundException`, but the Aliyun OSS input stream path appears to propagate Aliyun SDK `OSSException` for `NoSuchKey` directly from `client.getObject(...)`. That exception is not classified as Iceberg `NotFoundException`, so Iceberg's generic retry path keeps retrying.
By contrast, Iceberg's AWS S3 input stream maps `NoSuchKeyException` to `NotFoundException`, allowing the retry loop to stop quickly.
Relevant upstream source references:
- Iceberg retry stops on `NotFoundException`: `BaseMetastoreTableOperations.refreshFromMetadataLocation`
- Iceberg Aliyun OSS stream reads with `client.getObject(...)` without translating `NoSuchKey`: `OSSInputStream.openStream`
- Iceberg AWS S3 stream translates `NoSuchKeyException` to `NotFoundException`: `S3InputStream.openStream`
- Gravitino table operations hold tree locks around load/drop paths: `TableOperationDispatcher.loadTable` and `dropTable`
### Possible fixes
1. In Gravitino's Iceberg REST wrapper, classify Aliyun OSS `NoSuchKey`/`NoSuchBucket` failures from metadata loading as a not-found table condition instead of allowing generic retries to continue.
2. Add a regression test around Iceberg REST `loadTable`/`tableExists` behavior when metadata loading fails with Aliyun OSS `NoSuchKey`.
3. Longer term, consider avoiding heavyweight `loadTable` calls for shallow existence checks used by UI authorization/tag/policy flows.
I am preparing a small PR for the fast-fail behavior.
Contributor guide
Assessment
This issue has not been assessed yet.