apache / apache/gravitino

[FEATURE] Pluggable table location provider for the generic lakehouse catalog

Open
#12,429 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

Introduce a pluggable SPI that lets deployments delegate table location assignment and release to an external service, instead of requiring the location to be a statically configured, already-known path.
Today `GenericCatalogOperations#calculateTableLocation` resolves a managed table's location through a fixed, hard-coded chain:
1. the location table property (for Lance, this is what the `x-lance-table-location` REST header ends up as);
2. otherwise, the location schema property + table name;
3. otherwise, the location catalog property + schema name + table name;
4. otherwise, throw `IllegalArgumentException`.
The method is private and the chain is not extensible. Symmetrically, when a table is dropped nothing at all happens to the storage that was assigned to it.
We propose adding an interface — working name `TableLocationProvider` — discovered via `ServiceLoader` and selected by a catalog property, consulted when a managed table is created without an explicit location, and notified when such a table is dropped.

### Motivation

**Assignment.** All three existing sources share the same assumption: *someone already knows the
concrete path, and supplies it as configuration or as request input.*

In enterprise deployments that assumption often does not hold. Physical storage layout is owned by
a central data-platform service, and the path for a new table is *allocated* at creation time
rather than configured ahead of time. Allocation typically depends on inputs Gravitino has no
opinion about — owning team, organizational domain, region, storage tier, quota and capacity,
retention policy — and it usually has side effects, such as creating or reserving the backing
bucket/prefix and registering ownership for lifecycle and cost attribution.

Under the current design an operator has two options, both unsatisfying:

- **Push the problem to the client.** Every writer must call the platform's allocation service
itself and then pass the resulting path in via `x-lance-table-location` (or the `location`
property). This spreads a platform concern across every engine and every user, and nothing stops
a client from passing an arbitrary path that bypasses platform policy entirely.
- **Fork Gravitino.** Patch `calculateTableLocation` locally, which makes tracking upstream
painful.

A first-class extension point removes both, and lets the deployment *enforce* the policy
server-side rather than hoping clients cooperate.

**Release.** The gap on the drop side is arguably worse, because it silently leaks.
`ManagedTableOperations#purgeTable` states it plainly:

> For Gravitino managed tables, `purgeTable` is equivalent to `dropTable`. It only removes the
> table metadata from the entity store. Physical data deletion should be handled by the specific
> catalog implementation if needed.

For a deployment that assigns its own paths this is at most a housekeeping matter. For a
deployment backed by an external allocator it means storage is stranded permanently: the allocator
is never told the space can be reclaimed, so quota accounting, cost attribution and retention
policy all drift out of sync with reality, with no way to tell an intentionally-dropped table
apart from one that never existed.

Gravitino does have an event mechanism — `TableEventDispatcher` dispatches `DropTablePreEvent` /
`DropTableEvent` (and the `Purge*` equivalents), and `EventListenerPlugin` supports a `SYNC` mode —
so this is not a total blind spot. It is, however, not usable for reclamation as it stands:
`DropTableEvent` carries only the user, the `NameIdentifier` and an `isExists` flag. It does not
carry the table's `location` or its properties, and by the time the post-event fires the metadata
is already gone, so a listener knows *which* table disappeared but has no way to find out *which
storage* it occupied.

A listener could in principle work around this by hooking `DropTablePreEvent`, loading the table to
capture its location, and stashing it until the post-event arrives — but that pushes a shadow copy
of Gravitino's own metadata into every deployment, and leaves dangling state whenever the two
events do not pair up. More fundamentally, the event mechanism is observational: it cannot *supply*
a location on the create path, which is the other half of what we need. Assignment and release are
two ends of the same contract, and we would rather they be expressed as one.

We hit both while running the Lance REST catalog against an internal path-allocation service, but
nothing about the problem is Lance-specific — `calculateTableLocation`, `dropTable` and
`purgeTable` are all shared by every table format the generic lakehouse catalog serves.

### Describe the solution

A new SPI in the generic lakehouse catalog, deliberately kept free of any domain-specific fields:

```java
public interface TableLocationProvider extends Closeable {

/** Name used to select this provider via the catalog property. */
String name();

void initialize(Map catalogProperties);

/**
* Returns the location to use for the table being created, or
* {@code Optional.empty()} to fall back to Gravitino's built-in resolution.
*/
Optional provideTableLocation(TableLocationContext context);

/**
* Notifies the provider that a table whose location it assigned has been dropped,
* so the underlying storage can be released or scheduled for reclamation.
*
*

Invoked only after the table metadata has been successfully removed. See the
* ordering and failure-handling notes below.
*/
default void releaseTableLocation(TableReleaseContext context) {}
}
```

`TableLocationContext` carries only what Gravitino already knows, so implementations can pick out
whatever their backend needs (owner, domain, tier, …) from the property maps without Gravitino
having to model those concepts:

- `NameIdentifier` of the table (metalake / catalog / schema / table)
- table format (`lance`, …)
- the table, schema and catalog property maps

`TableReleaseContext` carries the same identifiers plus the dropped table's resolved `location` and
its properties, and a `purge` flag distinguishing `dropTable` from `purgeTable` so the
implementation can choose soft versus hard reclamation.

### Wiring

- Discovered with `ServiceLoader`, mirroring the existing `LakehouseTableDelegator` /
`LakehouseTableDelegatorFactory` pattern in this same module, and `CredentialProvider` in
`common`.
- Selected per catalog via a property, e.g. `table-location-provider = `.
- `provideTableLocation` is called from `calculateTableLocation` **only after** the explicit
table-level `location` check, so an explicitly requested location still wins.
- `releaseTableLocation` is called from `dropTable` and `purgeTable`.

### Drop semantics — the part we would most like feedback on

We propose the following, and want to be explicit that it is a deliberate trade-off rather than a
fully reliable protocol:

- **Ordering: after metadata removal, never before.** If the provider ran first and succeeded but
metadata removal then failed, the result would be a table that is still visible in Gravitino
whose storage has been reclaimed — silent data loss. Running afterwards means the worst case is
storage that outlives its table, which is detectable and recoverable by reconciliation. We think
leaking is strictly preferable to losing.
- **Failure handling: log and emit a metric; do not fail the drop.** By the time the provider is
invoked the metadata is already gone and cannot be rolled back, so propagating the failure would
report an error for an operation that did in fact take effect. The SPI javadoc should state
clearly that delivery is best-effort and that implementations must be able to reconcile
independently. Building reliable delivery (outbox, retry queue) would be a much larger change and
we do not propose it here.

If maintainers would rather have a fail-the-operation contract, or prefer that the callback be a
separate listener-style extension rather than a method on this interface, we are happy to go that
way instead — the ordering constraint above is the only part we feel strongly about.

### Backward compatibility

Fully preserved, by construction:

- no `table-location-provider` configured → the current three-step chain runs unchanged and no drop
callback fires;
- provider configured but `provideTableLocation` returns `Optional.empty()` → falls back to the
current chain;
- explicit table-level `location` present → the provider is never consulted;
- `releaseTableLocation` is a `default` no-op, so implementations that only care about assignment
need not implement it.

No existing behaviour changes for any deployment that does not opt in.

### Scope for a first iteration

- **In scope:** `createTable`, `dropTable`, `purgeTable`.
- **Out of scope:** `alterTable(RenameTable)`. A rename changes the table's identity but not its
physical path, and whether an allocator needs to know is very implementation-dependent; we would
rather not guess. The context objects are shaped so a rename callback can be added later without
breaking existing implementations.
- **Also out of scope:** filesets in the Hadoop catalog, which have a structurally similar problem.
Worth a separate issue if this pattern is accepted.

### Additional context

- **Relevant code:**
`catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java`
(`calculateTableLocation`, `dropTable`, `purgeTable`);
`core/src/main/java/org/apache/gravitino/catalog/ManagedTableOperations.java`
(`purgeTable` javadoc quoted above).
- **Existing `ServiceLoader`-based extension points** we would follow for consistency:
`LakehouseTableDelegatorFactory` (same module) and `CredentialProvider` /
`CredentialProviderFactory` (`common`, `core`).
- We searched existing issues and did not find one covering pluggable location assignment or
release; apologies if we missed a prior discussion.
- We can help with the design doc and contribute the implementation and tests

Contributor guide

Open the contributing guide

Research direction

Start with GenericCatalogOperations.java, especially calculateTableLocation, dropTable, and purgeTable, then compare the ServiceLoader patterns used by LakehouseTableDelegatorFactory and CredentialProvider. Read ManagedTableOperations.java for the existing purge semantics. Done means an agreed SPI design and coverage of create, drop, and purge behavior without changing deployments that do not configure a provider.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.