apache / apache/gravitino

[Improvement] Address TreeLock limitations for Gravitino HA deployment

Closed
#10,474 5 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

#### Background

Gravitino uses `TreeLock` (implemented in `core/src/main/java/org/apache/gravitino/lock/`) to ensure consistency and atomicity of metadata operations. The lock follows a hierarchical read-write locking strategy — it acquires read locks on all ancestor nodes and a write/read lock on the target node itself. For example, renaming a table `metalake.catalog.db.table1` acquires:

```
/ → readLock
/metalake → readLock
/metalake/catalog → readLock
/metalake/catalog/db → writeLock (parent of the renamed resource)
```

This design is managed by `LockManager`, which maintains an in-memory tree of `TreeLockNode`s backed by Java's `ReadWriteLock`.

#### Problem

`TreeLock` is **entirely in-process**. Each Gravitino server instance has its own `LockManager` with its own isolated lock tree in JVM memory. This fundamentally breaks down in **High Availability (HA) mode**, where multiple Gravitino server instances operate behind a load balancer:

1. **No cross-process coordination**: A write lock acquired on server A has zero visibility to server B. Two servers can concurrently modify the same metadata resource (e.g., both renaming the same table), causing data races and inconsistency.
2. **Race conditions on shared storage**: The backend metadata store (relational database) is shared across all nodes, but the locking layer protecting it is per-node, so conflicting writes can corrupt metadata state.
3. **Dead-lock detection is per-JVM**: The built-in dead-lock checker in `LockManager` only inspects threads within the local JVM; cross-node lock contention is entirely invisible.

#### Scope of Impact

`TreeLockUtils.doWithTreeLock(...)` is called across the majority of managers:

- `CatalogManager`, `MetalakeManager`
- `TableOperationDispatcher`, `SchemaOperationDispatcher`, `FilesetOperationDispatcher`, `TopicOperationDispatcher`, `ViewOperationDispatcher`, `ModelOperationDispatcher`, `PartitionOperationDispatcher`
- `TagManager`, `PolicyManager`, `StatisticManager`, `AccessControlManager`, `PermissionManager`, `OwnerManager`

Any metadata operation in these managers is potentially unsafe under concurrent HA traffic.

---

### How should we improve?

Two broad approaches are worth evaluating. Each has significant trade-offs:

**Option A — Distributed Lock**
Replace or wrap `TreeLock` with a distributed locking mechanism (e.g., ZooKeeper, etcd, or Redis-based). This preserves the existing hierarchical lock semantics and requires minimal changes to call sites.

- Pros: Semantically equivalent to current model; relatively bounded change surface.
- Cons: Introduces an external dependency; adds network round-trips on every metadata operation; requires handling distributed lock lease expiry, fencing tokens, and failure scenarios; potential new SPOF.

**Option B — Remove TreeLock; rely on storage-level consistency**
Eliminate `TreeLock` entirely and push concurrency control into the storage layer using optimistic locking (version-based CAS), database-level row locks, or serializable transactions.

- Pros: No external dependency; naturally correct in HA because all nodes share the same storage.
- Cons: Significant refactoring across all managers; requires retry logic for conflicts; may increase database load; harder to express coarse-grained parent-level write locks (e.g., "lock the whole schema while renaming a table").

**Option C....**

---

### Expected outcome

A concrete solution proposal covering:

1. Detailed analysis of the current `TreeLock` implementation and its limitations in HA mode.
2. Proposed solution with:
- Design overview and key components/interfaces to change.
- Analysis of correctness (does it fully eliminate the race conditions?).
- Analysis of performance impact.
- Migration strategy (how to transition from the current design without breaking existing behavior).
3. Optionally, a proof-of-concept implementation or pseudocode for the critical path.

Contributor guide

Open the contributing guide

Research direction

Start with core/src/main/java/org/apache/gravitino/lock/ and trace LockManager, TreeLock, and TreeLockUtils.doWithTreeLock(...) into the managers listed in the issue. Compare the current in-process locking behavior with HA requirements, then document a concrete design, correctness and performance analysis, migration strategy, and any proof of concept or pseudocode needed for the critical path.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.