[Bug report] Mixed list+create workload collapses: batchListEntitiesByRelation holds cache locks across the backend DB call
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main (also affects 1.3.x)
### Describe what's wrong
With authorization and the entity cache enabled, running `listTables` and `createTable` concurrently on a large schema makes creates very slow. Each workload on its own is fine — only the mix is affected.
Measured on a ~10k-table schema (weighted mixed load), `create_table` p50:
| Concurrency | expected (1.2) | observed |
| --- | --- | --- |
| 30 users | ~1.5s | 37s |
| 50 users | ~1.1s | 54s |
| 100 users | ~1.0s | 204s |
Under a heavier run (25 concurrent listers + 25 creators), creates stop completing entirely — they block server-side, waiting on a cache lock that the lists hold. Reads (`loadTable`) stay fast throughout.
### Root cause
`RelationalEntityStore.batchListEntitiesByRelation` runs the backend DB query **inside** `cache.withMultipleKeyCacheLock(...)`, so it holds the cache segment locks for the whole batch **across the DB round-trip**.
List authorization preloads owners for the entire page (`MetadataAuthzHelper.preloadOwner` → this method), which grabs most of the cache's segment locks for the full DB latency. Meanwhile `createTable` → `insertRelation` → `CaffeineEntityCache.invalidate` needs one of those locks, so every create waits for the list's entire DB call. Raising `gravitino.cache.lockSegments` does not help (a larger list just grabs proportionally more segments).
### How to reproduce
1. Enable authorization and the entity cache.
2. Create a schema with a large number of tables.
3. Run `listTables` and `createTable` concurrently against it.
4. `create_table` latency spikes (and hangs under sustained lists) while lists are in flight.
### Potential fix
Don't hold a cache lock across the DB call. Read cache-aside instead: check the cache lock-free, run the backend query with no lock held, then populate (each `put` takes its own short per-key lock). This keeps individual list/get paths correct while letting concurrent creates proceed.
Draft PR: #12171.
### Willingness to contribute
Yes — PR attached.
Contributor guide
Assessment
This issue has not been assessed yet.