apache / apache/gravitino

[Bug] Iceberg REST retries Aliyun OSS NoSuchKey and holds Gravitino tree locks for missing metadata files

Open
#11,039 2 comments 0 reactions 1 assignee Claimed by @atovk View on GitHub
bug
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What happened

When an Iceberg table registered in Gravitino points to an Aliyun OSS metadata JSON that no longer exists, Gravitino's Iceberg REST service can spend a long time retrying the missing object. During that time, Gravitino table/tag/policy requests may hold tree locks for more than 30 seconds, causing the UI and metadata APIs for the same schema to appear stuck.

### Why this matters

A single stale or externally damaged Iceberg table can degrade a whole schema's metadata browsing experience. This is especially visible in the web UI, which may issue table detail, tags, policies, and existence checks close together.

### Observed behavior

For a table similar to `metalake.catalog.schema.table`, where the JDBC Iceberg catalog row still pointed at a missing metadata file, the Gravitino server log repeatedly showed:

```text
Retrying task after failure: sleepTimeMs=... The specified key does not exist.
.../metadata/00000-...metadata.json
```

The same time window also showed tree-lock watchdog warnings similar to:

```text
Thread with identifier ThreadIdentifier{thread=Thread[Gravitino-webserver-...,5,main], ident=metalake.catalog.schema.table} holds the lock node TreeLockNode{ident=/,...} for more than 30s ... io-connection hangs
```

After unregistering/dropping the stale table entry, the same API returned quickly with `NoSuchTableException`/404.

### Expected behavior

Missing Aliyun OSS metadata objects should be treated as a non-retryable not-found condition and should fail quickly. A stale table should not cause long retry loops that hold Gravitino tree locks and block unrelated metadata browsing in the same schema.

### Root cause analysis

Gravitino 1.2.0 uses Iceberg 1.10.1. Iceberg's `BaseMetastoreTableOperations.refreshFromMetadataLocation` stops retrying on Iceberg `NotFoundException`, but the Aliyun OSS input stream path appears to propagate Aliyun SDK `OSSException` for `NoSuchKey` directly from `client.getObject(...)`. That exception is not classified as Iceberg `NotFoundException`, so Iceberg's generic retry path keeps retrying.

By contrast, Iceberg's AWS S3 input stream maps `NoSuchKeyException` to `NotFoundException`, allowing the retry loop to stop quickly.

Relevant upstream source references:

- Iceberg retry stops on `NotFoundException`: `BaseMetastoreTableOperations.refreshFromMetadataLocation`
- Iceberg Aliyun OSS stream reads with `client.getObject(...)` without translating `NoSuchKey`: `OSSInputStream.openStream`
- Iceberg AWS S3 stream translates `NoSuchKeyException` to `NotFoundException`: `S3InputStream.openStream`
- Gravitino table operations hold tree locks around load/drop paths: `TableOperationDispatcher.loadTable` and `dropTable`

### Possible fixes

1. In Gravitino's Iceberg REST wrapper, classify Aliyun OSS `NoSuchKey`/`NoSuchBucket` failures from metadata loading as a not-found table condition instead of allowing generic retries to continue.
2. Add a regression test around Iceberg REST `loadTable`/`tableExists` behavior when metadata loading fails with Aliyun OSS `NoSuchKey`.
3. Longer term, consider avoiding heavyweight `loadTable` calls for shallow existence checks used by UI authorization/tag/policy flows.

I am preparing a small PR for the fast-fail behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.