apache / apache/gravitino

[Subtask] Background reconcile to remove stale store rows for external-backed catalogs

Open
#12,155 0 comments 0 reactions 0 assignees View on GitHub
subtask
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the subtask

A Gravitino store row can outlive its external object — after a swallowed/failed `store.delete` (`TableOperationDispatcher.dropTable` ignores `NoSuchEntityException`), a crash between the external op and the store write, or an out-of-band drop in Hive/JDBC. Today such stale rows are only repaired lazily on the next read, and actively removed only for schemas (`OrphanedSchemaCleanup`). There is no general active cleanup.

Pre-existing gap, independent of the TreeLock work.

### How to fix

Generalize the existing `OrphanedSchemaCleanup` / `SchemaEntityCleaner` pattern (external-existence probe -> delete orphaned store rows) from schemas to table/fileset/topic/model. Reuse `deleteTable`'s cascade so relations are removed with the row.

Guardrails (an external probe is slow and can flap):

- External-backed catalogs only — never managed catalogs, whose store is authoritative.
- Delete only after **N consecutive misses + a grace period**, to survive transient external outages.
- Rate-limit and run on a low-priority schedule (the `RelationalGarbageCollector` cadence fits).
- Tests: a stale row with the external object gone is reconciled; a transient external outage does **not** delete a live row.

### Parent issue

https://github.com/apache/gravitino/issues/10238

Contributor guide

Open the contributing guide

Research direction

Start by reading OrphanedSchemaCleanup and SchemaEntityCleaner, then inspect TableOperationDispatcher.dropTable and the RelationalGarbageCollector cadence. Define reconciliation for external-backed table, fileset, topic, and model rows using consecutive misses, a grace period, and rate limiting. Done means a missing external object is cleaned with its relations, while a transient outage leaves a live row intact; add tests for both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.