apache / apache/gravitino

[Subtask] Make the caffeine entity cache multi-node correct via cross-node change-log invalidation

Open
#12,019 1 comment 0 reactions 0 assignees View on GitHub
subtask
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 15h
Merged PRs (30d)
315

Description

### Describe the subtask

Today the entity store cache only invalidates on the node that made the change, so other nodes keep serving stale data until their entries expire. This subtask makes the default `caffeine` cache correct across nodes by reusing the existing `entity_change_log` and `EntityChangeLogPoller` that the jcasbin and catalog caches already use, so no new infrastructure is added.

It also adds a coherence marker to the cache SPI so the write path can tell a per-node cache (which must be told about changes made on other nodes) from a shared cache (which does not). The later Redis implementation relies on the same marker.

This subtask should:

- Add `EntityCache.coherence()` returning `LOCAL_PER_NODE | SHARED`, implement `LOCAL_PER_NODE` for `CaffeineEntityCache`, and add the write-path coherence gate.
- Add an entity-store `EntityChangeLogListener` that replays each ALTER/DROP row as an `invalidate(ident, type)`, with a container drop cascading to cached children through the existing forward prefix scan (`cacheIndex`).
- Register the listener as a third consumer of the existing poller, next to the catalog cache and the jcasbin id-mapping cache; the structural entities (metalake, catalog, schema, table, topic, view, fileset, model) already emit rows.
- Add a change-log emit point for **tag and policy** (their MetaServices emit no row today), written in the same transaction as the tag/policy write, so a per-node cache can refresh them across nodes.
- Reuse the lag-safe, id-based polling built for the change log (see #11736) and keep the cache TTL as a backstop.
- Add multi-node integration tests: an ALTER/DROP on node A is reflected on node B within one poll interval (table / schema / catalog and tag / policy), a schema drop clears the child tables on the other node, and adding the new consumer does not change the two existing ones.

This is the main Phase 1 deliverable of #11737: `gravitino.cache.enabled=true` becomes safe in multi-node deployments with the default cache.

### Parent issue

https://github.com/apache/gravitino/issues/11737

Contributor guide

Open the contributing guide

Research direction

Start with the EntityCache and CaffeineEntityCache SPI, then trace EntityChangeLogPoller consumers for the catalog and jcasbin caches. Follow the entity-store write paths and existing cacheIndex prefix scan before adding the listener and tag/policy change-log emissions. Done means multi-node integration tests show ALTER/DROP invalidation on the other node within one poll interval without changing the existing consumers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.