[Performance] Optimize external metadata cache incremental update costs

### Background

This issue is a follow-up for PR #65126.

PR #65126 refactors the external metadata names cache to use immutable snapshots with copy-on-write style updates. That change was introduced for correctness rather than performance.

The main goal of this refactor is to make name-cache publication and invalidation semantics safe under concurrent loads and incremental HMS events. In particular, the PR needs a names-cache representation that can:
- publish a consistent snapshot to readers
- avoid in-place mutation of shared cache state
- cooperate with generation-based invalidation / stale-load fencing
- prevent lagging async/manual loads from overwriting newer HMS event updates

Using immutable snapshots helps make these correctness guarantees explicit and avoids readers observing partially updated name/index state during concurrent cache refresh or event handling.

At the same time, this refactor also changes the HMS event update path from the previous incremental-style update to a full snapshot rebuild in some hot-name-cache cases. As a result, it introduces a real performance regression on HMS CREATE/DROP event handling for very large metadata sets.

This follow-up is only relevant if PR #65126 is merged.

### Problem

On HMS table/database name events, the current update path may rebuild the full names snapshot for a single name change. In large external catalogs, one CREATE or DROP event may:

- scan the full names list
- copy the full list
- recreate existing entries
- rebuild derived indexes and maps

This makes a single HMS event cost O(n) in the size of the hot names snapshot.

More specifically, `NameCacheValue.withName()` currently:

1. scans the existing snapshot for conflicting remote-name mappings
2. copies the full names list
3. scans the copy to remove an existing local-name entry
4. recreates every `Pair` while constructing the new snapshot
5. rebuilds the immutable names list, local names list, lower-case index, local-to-remote index, and local-name set

`withoutLocalName()` has the same full-copy and full-index-rebuild behavior after its removal scan.

### Comparison with the master baseline

The master implementation stores a mutable `List>` directly as the Caffeine names-cache value. Incremental CREATE/register modifies it in place with `v.add(...)`, while DROP/unregister uses an in-place `removeIf(...)`.

This means the cost regression is not identical for every event:

- **CREATE/register:** master is approximately O(1) for the names update, while PR #65126 changes it to an O(n) snapshot rebuild. This is the primary performance regression.
- **DROP/unregister:** master is already O(n) because of `removeIf`, but it mutates the existing list. PR #65126 remains O(n) while adding full-list copying, entry recreation, and derived-index rebuilding, so the asymptotic complexity is unchanged but CPU and allocation constants are higher.
- **Rename:** an unregister/register sequence may rebuild the snapshot twice.
- **Name lookup:** the trade-off is not uniformly negative. Some master lookups scan the names list in O(n), while the new snapshot provides O(1) derived indexes.

### Why reverting to the master mutation model is not acceptable

The master implementation is cheaper for incremental writes, but it has important concurrency and consistency weaknesses:

1. **Readers and event handlers share the same mutable list**
- `listNames()` can iterate the same list that an HMS event modifies with `add` or `removeIf`.
- Caffeine serializes a map mutation for the cache key, but it does not make an escaped mutable value immutable or force readers holding that value to use the same lock.
- Readers may observe an unstable snapshot or encounter concurrent-modification behavior.

2. **In-flight loads are not protected by a generation fence**
- A names load or refresh that started before an HMS event is not explicitly rejected when it completes after the event.
- This can let a lagging result overwrite or reintroduce state that a newer CREATE/DROP/rename event has already updated.

3. **The names list and derived lookup maps are published separately**
- The master path maintains the names list and `lowerCaseToTableName`/`lowerCaseToDatabaseName` as separate mutable structures.
- During incremental updates or full reloads, a reader can observe a new names list with an old/partial lookup map, or the reverse.

4. **Incremental CREATE is not naturally idempotent**
- The master update path appends the new pair without first replacing an existing local-name entry.
- Duplicate, retried, or replayed events can therefore create duplicate name entries or order-dependent lookup behavior.

These weaknesses can result in inconsistent name resolution, stale metadata being republished after an event, duplicate list entries, or transient failures during concurrent name enumeration. They are correctness concerns rather than only performance concerns.

### Correctness benefits provided by PR #65126

The immutable snapshot design in PR #65126 provides:

- atomic publication: readers see either the complete old snapshot or the complete new snapshot
- derived indexes that are built and published together with the names list
- generation-based fencing for stale in-flight loads
- idempotent local-name replacement in incremental updates
- stable cache-only replay and invalidation behavior
- O(1) remote/local and case-insensitive name lookup through prebuilt indexes

The follow-up optimization must preserve these properties. It should not restore in-place mutation of a shared names list merely to recover the master's incremental-write cost.

### Impact

This is a performance regression introduced by PR #65126, not a correctness blocker.

The impact is most visible on the HMS event path with very large metadata sets. In normal usage, CREATE/DROP is usually not a high-frequency operation, and most users will not have extremely large numbers of databases or tables under a single catalog/database. The problem is therefore concentrated in large-scale HMS event scenarios.

For example, if a hot database snapshot contains 100,000 table names, adding one table may traverse the full snapshot several times and recreate approximately 100,000 name entries plus multiple indexes. Repeated DDL or event bursts can therefore increase FE CPU usage, allocation rate, GC pressure, and HMS event processing latency.

Also, this copy-on-write overhead has already been discussed in earlier review rounds, and parts of the PR series have already reduced the impact on more frequent execution paths. The remaining concern is mainly the HMS event path.

### Why this is tracked separately

By the current review stage of PR #65126, the remaining issues are already being narrowed down progressively. Fixing this regression properly likely requires a broader redesign of the names-cache update/publication strategy, which would significantly expand scope and increase late-stage rework risk.

So instead of mixing that architectural change into the current PR, this issue tracks the follow-up performance fix after PR #65126 is merged, or together with other known HMS event follow-up work.

### Detailed design proposal

The recommended follow-up design is documented here:

- [External Metadata Name Cache: Immutable Base + Bounded Delta Overlay Design](https://gist.github.com/wenzhenghu/33a7eb917f6814fc915f73fd4838be0b)

The proposal keeps `NameCacheValue` as the single immutable value published through the existing names-cache key, retains the current `MetaCacheEntry` generation/publication protocol, and replaces per-event full snapshot rebuilds with a bounded immutable delta plus synchronous threshold-based compaction. It also defines ordering, mapping, case-insensitive lookup, idempotency, testing, benchmarking, change scope, and rollout constraints.

### Candidate directions

Possible directions under consideration:

1. **Stripe/shard the names cache**
- split the names snapshot/index into multiple stripes
- limit copy/rebuild work to the affected subset for each event
- preserve the current snapshot visibility guarantees

2. **Control deep-copy vs reference reuse**
- add an internal mechanism to choose whether immutable entries are deep-copied or directly reused
- reduce redundant object recreation and index rebuild work
- keep publication safety and immutability guarantees explicit

3. **Use structural sharing or a bounded delta layer**
- represent single-name changes without recreating every unchanged entry
- compact accumulated deltas into a new base snapshot at a controlled threshold
- keep a clearly defined atomic publication boundary for readers

4. **Batch adjacent HMS name events**
- coalesce multiple CREATE/DROP updates
- publish one rebuilt snapshot for an event batch instead of one per event

These are candidate ideas only and still need design validation.

### Follow-up expectations

The follow-up should:
- reduce the O(n) rebuild cost for ordinary single-name HMS events
- preserve immutable/atomic reader-visible snapshots
- preserve generation-based stale-load fencing
- keep the names list and all derived indexes mutually consistent
- retain idempotent behavior for duplicate or replayed events
- include targeted benchmark or regression coverage for large hot snapshots and event bursts
- clearly document the invariants of the final cache design

Read PR #65126 and the linked Immutable Base + Bounded Delta Overlay design first, then inspect NameCacheValue.withName(), withoutLocalName(), and the HMS event update path. Compare candidate designs against the stated publication and generation-fencing invariants. Done means reducing ordinary single-name rebuild cost while preserving atomic snapshots, consistent indexes, idempotency, and targeted large-snapshot or event-burst benchmark coverage.

Written by the indexing model from the issue text.