apache / apache/gravitino

[Subtask] Design doc: TreeLock necessity and concurrency control for multi-node Gravitino

Open
#12,156 0 comments 0 reactions 0 assignees View on GitHub
subtask
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Describe the subtask

Write and land the design document that analyzes whether TreeLock is needed once Gravitino runs in HA, and decides the direction. It should cover:

- **Background & problem**: TreeLock is per-JVM, so it protects nothing across nodes behind a load balancer.
- **Analysis**: every catalog write touches two stores (external catalog + Gravitino store) with no shared transaction; for external-backed catalogs the external system is already the cross-node judge, while managed catalogs (fileset/model) rely only on the Gravitino store.
- **Two directions compared**: (1) remove the lock, move correctness into the shared DB (OCC + conditional insert); (2) keep a cross-node lock (external service / DB `FOR UPDATE`).
- **Industry survey**: how comparable systems (Iceberg REST/Polaris, Nessie, Glue, Metacat, Kubernetes, Debezium outbox, Hive ACID) keep two stores consistent.
- **Conclusion**: choose Direction 1; shrink TreeLock only after the DB rules land.
- **Proposal & task breakdown**: Changes 1-4 plus the pre-existing hazards split out as do-first sub-issues (#12153, #12154, #12155).

Draft: `design-docs/treelock-necessity-and-concurrency-design.md` (with rendered diagrams under `design-docs/images/`).

### Parent issue

https://github.com/apache/gravitino/issues/10238

Contributor guide

Open the contributing guide

Research direction

Start with design-docs/treelock-necessity-and-concurrency-design.md and inspect the rendered diagrams under design-docs/images/. Compare the two stated concurrency directions and the industry examples, then ensure the document covers the requested analysis, chooses Direction 1, and includes Changes 1–4 with hazards #12153, #12154, and #12155 split out.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
databases, distributed-systems, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.