apache / apache/gravitino

[Subtask] Add metrics and diagnostic logs for entity change log processing

Open
#12,419 0 comments 0 reactions 1 assignee Claimed by @yuqi1129 View on GitHub
subtask
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the subtask

When a stale-cache incident occurs in a multi-node deployment, operators need to determine whether:

1. the writer appended an `entity_change_log` row;
2. the peer poller observed the row;
3. every listener consumed the batch; and
4. the local cache invalidated the expected key (or fell back to a full clear).

`EntityChangeLogPoller` already has lifecycle, batch, cursor, and retry logs, and `EntityCacheChangeLogListener` has a basic invalidation log. However, the poller still exposes no metrics, the write path has no diagnostic log, and the existing messages do not carry consistent fields that can be correlated across the normal insert -> poll -> deliver -> invalidate path.

### Metrics

Expose the following through the existing `MetricsSource` infrastructure, using stable names under an entity-change-log prefix:

- **DB tail ID** and **current cursor** (`entityPollHighWaterId`).
- **record lag**: `max(entity_change_log.id) - entityPollHighWaterId`, sampled by the poll cycle rather than executing a DB query from a gauge callback.
- **time since the last successful DB poll**. An empty result is still a successful poll.
- **pending delivery state**: whether `pendingDelivery` is non-null, its attempt number, and the number of pending listeners.
- **poll failures** and **listener failures**, with listener failures attributable to a stable listener name.
- **records fetched/applied**, **batch size**, and **poll duration**.
- **targeted invalidation failures / fallback full-cache clears**.

The gauges must be safe to read concurrently with the scheduler thread and must not perform blocking database work when JMX or Prometheus scrapes them.

### Operational logging

Add or retain logs for the complete daily path, with consistent `key=value` fields where applicable:

- **Write (`DEBUG`)**: after an `entity_change_log` insert succeeds, log `metalake`, `entityType`, `operateType`, and encoded `fullName`. Make it clear that the row was appended to the current transaction; do not claim that an outer transaction has committed.
- **Poll (`DEBUG`)**: log the input cursor, fetched count, ID range, duration, and a bounded record summary for non-empty batches. Empty polls must not produce `INFO` noise.
- **Delivery (`DEBUG`)**: log the listener name, batch ID range, record count, and retry attempt. Keep `WARN`/`ERROR` logs for query failures, listener failures, a paused cursor, exhausted retries, and the configured `EXIT`/`SKIP` action.
- **Invalidate (`DEBUG`)**: for each targeted invalidation, include `changeId`, `entityType`, `operateType`, decoded identifier, and encoded `fullName`. Log a batch completion summary with applied/skipped counts and duration.
- **Fallback (`ERROR`)**: if targeted invalidation fails, include the failing change context and explicitly record the full local-cache clear. Malformed rows must include `changeId` and the invalid field in `WARN` logs.
- **Cursor/lifecycle (`INFO`)**: retain startup/shutdown state and successful cursor advancement with old/new cursor, batch range, batch size, and approximate age.

Do not log entity payloads, catalog properties, credentials, or other secrets. Detailed success-path messages should remain at `DEBUG`; `INFO` should be limited to lifecycle and meaningful state transitions.

### Acceptance criteria

- The metrics are visible through both JMX and Prometheus with documented, stable names and units.
- Unit tests cover metric state transitions for an empty successful poll, a non-empty poll, a DB failure, listener retry/recovery, and exhausted retries.
- Unit tests cover the key write and invalidate diagnostic contexts, including malformed rows and fallback cache clearing, without depending on an entire formatted log line.
- In a two-node run, logs can show the sequence from a write on node A to polling, listener delivery, and targeted invalidation on node B using the entity fields and change-log ID.
- Logging does not add a DB query, transaction boundary, or cache mutation to the success path beyond the work required for the metrics themselves.

### Parent issue

https://github.com/apache/gravitino/issues/11737

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.