apache / apache/gravitino

[Bug report] Mixed list+create workload collapses: batchListEntitiesByRelation holds cache locks across the backend DB call

Open
#12,170 5 comments 0 reactions 2 assignees Claimed by @bharos View on GitHub
2.0.0 bug cache
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Version

main (also affects 1.3.x)

### Describe what's wrong

With authorization and the entity cache enabled, running `listTables` and `createTable` concurrently on a large schema makes creates very slow. Each workload on its own is fine — only the mix is affected.

Measured on a ~10k-table schema (weighted mixed load), `create_table` p50:

| Concurrency | expected (1.2) | observed |
| --- | --- | --- |
| 30 users | ~1.5s | 37s |
| 50 users | ~1.1s | 54s |
| 100 users | ~1.0s | 204s |

Under a heavier run (25 concurrent listers + 25 creators), creates stop completing entirely — they block server-side, waiting on a cache lock that the lists hold. Reads (`loadTable`) stay fast throughout.

### Root cause

`RelationalEntityStore.batchListEntitiesByRelation` runs the backend DB query **inside** `cache.withMultipleKeyCacheLock(...)`, so it holds the cache segment locks for the whole batch **across the DB round-trip**.

List authorization preloads owners for the entire page (`MetadataAuthzHelper.preloadOwner` → this method), which grabs most of the cache's segment locks for the full DB latency. Meanwhile `createTable` → `insertRelation` → `CaffeineEntityCache.invalidate` needs one of those locks, so every create waits for the list's entire DB call. Raising `gravitino.cache.lockSegments` does not help (a larger list just grabs proportionally more segments).

### How to reproduce

1. Enable authorization and the entity cache.
2. Create a schema with a large number of tables.
3. Run `listTables` and `createTable` concurrently against it.
4. `create_table` latency spikes (and hangs under sustained lists) while lists are in flight.

### Potential fix

Don't hold a cache lock across the DB call. Read cache-aside instead: check the cache lock-free, run the backend query with no lock held, then populate (each `put` takes its own short per-key lock). This keeps individual list/get paths correct while letting concurrent creates proceed.

Draft PR: #12171.

### Willingness to contribute

Yes — PR attached.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.