[Enhancement] Include catalog and database MetaCacheEntry instances in unified cache statistics
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Status after connector SPI merge (#64304)
This issue remains open after #64304 and its scope is now broader.
The unified FE endpoint still reads only engine caches registered in ExternalMetaCacheMgr. It does not include:
1. the four direct FE MetaCacheEntry categories owned by ExternalCatalog and ExternalDatabase; or
2. connector-owned MetaCacheEntry instances introduced by #64304, such as HMS table/partition caches and connector table/snapshot/file-listing caches.
The connector cache framework exposes local statistics, but the Connector SPI currently has no cache-statistics reporting surface through which fe-core can collect them. Therefore #64304 did not fix this issue; it added another cache ownership domain that the eventual solution should cover.
A follow-up implementation should remain cache-only and should enumerate only already initialized/built connector state. Statistics collection must not initialize a catalog, construct a connector sibling, perform remote I/O, or warm any cache.
## Search before asking
I searched existing Apache Doris issues for external metadata cache statistics and ExternalMetaCacheMgr.getCatalogCacheStats. The closed tracking issue #60686 lists unified monitoring as a framework goal, but I did not find a dedicated issue covering this remaining gap.
## Background
PR #65126 migrates the ExternalCatalog and ExternalDatabase name/object caches to MetaCacheEntry. Four cache entries are owned directly by the catalog hierarchy rather than by an engine cache registered in ExternalMetaCacheMgr:
- ExternalCatalog database names
- ExternalCatalog database objects
- ExternalDatabase table names
- ExternalDatabase table objects
MetaCacheEntry exposes statistics for these entries, but ExternalMetaCacheMgr.getCatalogCacheStats(catalogId) currently reports only engine caches registered in its catalog cache registry. As a result, the unified statistics endpoint omits the central catalog/database name and object caches.
## Problem
Operators cannot observe capacity, estimated size, hits, misses, load success/failure, eviction, invalidation, or the last load error for these four cache categories through the unified per-catalog statistics path. This makes cache sizing and cache-consistency diagnosis incomplete even after the MetaCacheEntry migration.
## Expected behavior
ExternalMetaCacheMgr.getCatalogCacheStats(catalogId), or the public statistics API built on it, should include both:
1. Engine metadata caches registered in ExternalMetaCacheMgr.
2. Direct MetaCacheEntry instances owned by ExternalCatalog and its ExternalDatabase objects.
Statistics collection must remain cache-only: it must not initialize a catalog, enumerate remote metadata, load a database/table object, or turn a cold cache hot merely to produce statistics.
Cache names should be stable and distinguish at least catalog database-name/object entries from database table-name/object entries. If table entries are aggregated across databases, the aggregation semantics should be documented; otherwise each database should have a stable owner-qualified cache name.
## Suggested implementation
- Add a catalog-side statistics collector for the direct MetaCacheEntry instances.
- Let ExternalMetaCacheMgr merge those statistics with the registered engine-cache statistics.
- Traverse only already initialized/local catalog and database state.
- Preserve lifecycle safety for refresh, unregister, and replay paths.
## Suggested tests
- An uninitialized catalog returns statistics without triggering initialization or a remote call.
- Initialized but cold entries are reported with zero size and remain cold.
- Hot database-name/object and table-name/object entries report their own counters and sizes.
- Multiple databases are represented or aggregated deterministically.
- Refresh/reset/unregister removes stale owners from the statistics result.
## Related
- #60686
- #65126
Contributor guide
Research direction
Start with ExternalMetaCacheMgr.getCatalogCacheStats(catalogId), then read the direct MetaCacheEntry ownership in ExternalCatalog and ExternalDatabase and the connector cache statistics surface from #64304. Add cache-only collection for initialized local state and merge it with registered engine-cache statistics. Done means cold or uninitialized state stays cold, counters and sizes are reported with stable names, and refresh, reset, and unregister do not leave stale owners.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, databases, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100