[Subtask] Design doc: TreeLock necessity and concurrency control for multi-node Gravitino
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Describe the subtask
Write and land the design document that analyzes whether TreeLock is needed once Gravitino runs in HA, and decides the direction. It should cover:
- **Background & problem**: TreeLock is per-JVM, so it protects nothing across nodes behind a load balancer.
- **Analysis**: every catalog write touches two stores (external catalog + Gravitino store) with no shared transaction; for external-backed catalogs the external system is already the cross-node judge, while managed catalogs (fileset/model) rely only on the Gravitino store.
- **Two directions compared**: (1) remove the lock, move correctness into the shared DB (OCC + conditional insert); (2) keep a cross-node lock (external service / DB `FOR UPDATE`).
- **Industry survey**: how comparable systems (Iceberg REST/Polaris, Nessie, Glue, Metacat, Kubernetes, Debezium outbox, Hive ACID) keep two stores consistent.
- **Conclusion**: choose Direction 1; shrink TreeLock only after the DB rules land.
- **Proposal & task breakdown**: Changes 1-4 plus the pre-existing hazards split out as do-first sub-issues (#12153, #12154, #12155).
Draft: `design-docs/treelock-necessity-and-concurrency-design.md` (with rendered diagrams under `design-docs/images/`).
### Parent issue
https://github.com/apache/gravitino/issues/10238
Contributor guide
Research direction
Start with design-docs/treelock-necessity-and-concurrency-design.md and inspect the rendered diagrams under design-docs/images/. Compare the two stated concurrency directions and the industry examples, then ensure the document covers the requested analysis, chooses Direction 1, and includes Changes 1–4 with hazards #12153, #12154, and #12155 split out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- databases, distributed-systems, documentation
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100