[Bug] HMS incremental events have multiple metadata cache consistency gaps

## Status after connector SPI merge (#64304)

Updated against master after connector SPI PR #64304 was merged and PR #65126 was rebased on top of it.

| Item | Current status after #64304 | Tracking decision |
|---|---|---|
| 1. Rename cancelled by load-through existence check | Fixed by the current revision of #65126, pending merge | Keep in #65126 |
| 2. Partition event stale-load/lower-cache gap | Fixed structurally by #64304 | Partition events now call connector-owned partition invalidation; partition-name, partition-object, and file-listing caches are invalidated, and the connector cache generation fence rejects pre-event manual loads |
| 3. CREATE event bypasses visibility filters | Open; still present on current master and #65126 | Keep in this issue |
| 4. DROP_PARTITION writes a database deletion to ExternalMetaIdMgr | Obsolete/fixed by #64304 | The legacy event ID-mapping payload path was removed; event sync now persists only the cursor |
| 5. Database event ID uses the remote rather than canonical local name | Open; still present on current master and #65126 | Keep in this issue |
| 6. Database event publishes a non-canonical local key | Open; still present on current master and #65126 | Keep in this issue |
| 7. Database/table CREATE, DROP, and RENAME events do not invalidate connector-owned caches | Open; introduced by the cache-ownership split in #64304 and now part of master baseline | Track here rather than block #65126, provided #65126 does not claim connector-cache convergence |

No top-level issue is closed by this update because items 3, 5, 6, and 7 remain open.

## Search before asking

I searched existing Apache Doris issues and did not find an issue covering these HMS incremental-event cache consistency problems.

## Version

Current status verified on Apache Doris `master` at `af6dcff905114d7e05a6101496f5a629d8831b64`, after connector SPI PR #64304 was merged. The original report was collected at `3a75387e61388bd886eeef38b794d5ed4eb298bf`.

The remaining open items are master-baseline issues and are not introduced by #65126. Item 1 is fixed by #65126 pending merge; items 2 and 4 were fixed or eliminated by #64304.

## What's Wrong?

The original report identified multiple independent correctness gaps in HMS incremental event handling. The status table above reflects the current code after #64304.

### 1. ALTER TABLE/ALTER DATABASE rename can be incorrectly cancelled by a load-through "local existence" check

**Status: Fixed by the current revision of #65126 (pending merge).**

`AlterTableEvent.processRename()` calls `externalTableExistInLocal()` before applying the rename:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/AlterTableEvent.java#L107-L123

However, `HMSExternalCatalog.tableExistInLocal()` eventually calls the normal load-through table lookup rather than a cache-only lookup:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java#L190-L198

After HMS has renamed `old_table` to `new_table`, querying `new_table` from this check can load it from HMS and return `true`. The event is then cancelled, and cached state for `old_table` may remain. `AlterDatabaseEvent.processRename()` has the same structural problem through `catalog.getDbNullable(dbAfter.getName())`, although database rename is less commonly reachable in standard Hive.

The current revision of #65126 removes the load-through target-existence checks and makes both rename handlers converge local state by unregistering the old identity and registering the new identity. Its focused FE unit tests cover cold and already-hot rename targets.

### 2. ADD/DROP PARTITION events do not fence an in-flight partition-values load, and DROP may skip lower-level cache invalidation

**Current status: Fixed structurally by #64304. The details below describe the former legacy implementation.**

Both `addPartitionsCache()` and `dropPartitionsCache()` return immediately when the partition-values entry is absent:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java#L707-L746

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java#L748-L789

This causes two problems:

* If a partition-values load started before the event, the event does not invalidate/bump the entry, so the pre-event snapshot may still be published afterward.
* If the partition-values entry has been evicted while partition metadata or file listings remain cached, a DROP PARTITION event returns before invalidating those lower-level entries.

The second case can expose stale data when a partition name is later recreated with a different path or files. It also prevents the cache-miss fallback added by #65334 from being reached through this DROP event path.

### 3. Incremental CREATE_DATABASE/CREATE_TABLE events can bypass include/exclude visibility filters

Full names loading applies `include_database_list`, `exclude_database_list`, and `include_table_list`:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L551-L616

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L343-L356

The incremental register paths directly update the legacy metadata cache without applying the same visibility policy:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java#L205-L216

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java#L649-L663

When a names cache is already hot, an HMS CREATE event can therefore add a database/table that should remain hidden. The object can subsequently become queryable through the cached name.

### 4. DROP_PARTITION records a DATABASE deletion in ExternalMetaIdMgr

**Current status: Obsolete/fixed by #64304. The legacy per-event ID-mapping payload path no longer exists.**

`DropPartitionEvent.transferToMetaIdMappings()` uses `META_OBJECT_TYPE_DATABASE` instead of `META_OBJECT_TYPE_PARTITION`:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/DropPartitionEvent.java#L142-L152

`ExternalMetaIdMgr` interprets this as a request to remove the entire database mapping subtree:

https://github.com/apache/doris/blob/3a75387e61388bd886eeef38b794d5ed4eb298bf/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java#L145-L169

Current production use of the ExternalMetaIdMgr lookup APIs is limited, so the immediate query impact appears limited, but the persisted/replayed external metadata ID state is incorrect.

### 5. ALTER_DATABASE rename can generate an ID from the remote name instead of the canonical local name

**Status: Open. This is present on the master baseline and is not introduced by #65126.**

`CatalogMgr.registerExternalDatabaseFromEvent()` generates the event-created database ID directly from the incoming remote `dbName`:

https://github.com/apache/doris/blob/54e0e9712b6057d5f1c643103d1612d196b3cd35/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java#L772-L789

Normal external-database object loading instead generates the ID from the canonical `localDbName`:

https://github.com/apache/doris/blob/54e0e9712b6057d5f1c643103d1612d196b3cd35/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L459-L466

When `lower_case_database_names = 1`, or more generally whenever the incoming remote name differs from its canonical local cache key, the same database can therefore have two deterministic IDs:

* Immediately after the event: `ID(remote name) -> canonical local name`.
* After cache reset or normal reload: `ID(canonical local name) -> canonical local name`.

This makes the database ID and the local ID-to-name index unstable across refresh/reload and can break ID-based lookup consistency.

## What You Expected?

* Rename events should inspect only locally published cache state and should always remove the old local name. This is fixed by the current revision of #65126.
* Partition events should prevent pre-event snapshots from being published and should invalidate partition/file caches even when partition-values cache state is cold.
* Incremental create events should apply the same visibility rules as full names loading.
* DROP_PARTITION should delete only the corresponding partition metadata-ID mapping.
* Incremental database registration should derive the database ID from the same canonical local name used by normal object loading.
* Incremental database registration should also publish names/object/ID navigation state under that same canonical local cache key.

## How to Reproduce?

### Rename cancellation (fixed by the current revision of #65126)

1. Enable HMS incremental event synchronization.
2. Ensure the target name is not locally cached.
3. Rename `old_table` to `new_table` in Hive.
4. Process the ALTER_TABLE event.
5. On the affected baseline, the existence check loads `new_table` from HMS and cancels the event; cached `old_table` state may remain.

### Partition cache

1. Cache partition metadata/file listings for `p=1`.
2. Evict or invalidate only the table's partition-values entry, or block an in-flight partition-values load.
3. Drop `p=1` through HMS and process the DROP_PARTITION event.
4. Recreate `p=1` with a different location/files.
5. The old partition metadata or file listing may still be reused.

### Visibility filter

1. Create an HMS catalog with an include/exclude database filter or `include_table_list`.
2. Warm the corresponding names cache.
3. Create a filtered-out database/table directly in Hive.
4. Process the HMS CREATE event.
5. Observe that the filtered object is inserted into the cached names and becomes visible/queryable.

### External metadata ID mapping

1. Add a database/table/partition mapping to `ExternalMetaIdMgr`.
2. Process the partition's DROP_PARTITION mapping.
3. Observe that the database mapping subtree is removed instead of only the partition mapping.

### Database ID stability

1. Configure an HMS catalog with `lower_case_database_names = 1`.
2. Process an ALTER_DATABASE rename event whose new remote name contains uppercase characters.
3. Observe that `registerExternalDatabaseFromEvent()` generates the new ID from the remote mixed-case name while the local cache key is lowercase.
4. Reset or reload the catalog so the normal database-object loader runs.
5. Observe that the reloaded database ID is generated from the lowercase local name and differs from the event-created ID.

### Database local-key publication

1. Configure an HMS catalog with `lower_case_database_names = 1`.
2. Process a CREATE_DATABASE or ALTER_DATABASE rename event whose remote name contains uppercase characters.
3. Observe that the event-created database is published under the raw-case local key returned by `buildDbForInit(remoteDbName, null, ...)` rather than the canonical lowercase local key.
4. Perform a normal database lookup that first canonicalizes the name to lowercase.
5. Observe that the event-published database state no longer follows the same local-key convention as ordinary lookup and reload paths.

## Suggested Fixes

1. Fixed in the current revision of #65126: remove the load-through rename target-existence checks and always converge the old/new local identities.
2. For ADD/DROP PARTITION, invalidate or bump the partition-values key when it is cold/loading. DROP should unconditionally invalidate partition metadata/file cache entries for each dropped partition before optionally updating a hot partition-values snapshot.
3. Reuse a shared database/table visibility predicate in both full names loading and incremental registration.
4. Change the DROP_PARTITION mapping type to `META_OBJECT_TYPE_PARTITION` and add an event mapping unit test.
5. Resolve the canonical local database name with a cache-only helper, and use it consistently for event registration, invalidation, and `Util.genIdByName()`. Add mode-1 coverage that asserts the event-created ID equals the normal reload ID.
6. Reuse that same canonical local database name when publishing the event-created database into the names snapshot, object cache, and `dbIdToName` map. Add mode-1 coverage that asserts event-created cache keys match the normal lookup/reload convention.

The remaining four open fixes can be implemented as separate PRs if preferred.

### 6. Incremental database event registration can publish a non-canonical local cache key

**Status: Open. This is present on the master baseline and is not introduced by #65126.**

`HMSExternalCatalog.registerDatabase()` builds the event-created database object through `buildDbForInit(remoteDbName, null, ...)`:

https://github.com/apache/doris/blob/54e0e9712b6057d5f1c643103d1612d196b3cd35/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java#L196-L205

However, `ExternalCatalog.buildDbForInit()` only applies `fromRemoteDatabaseName(remoteDbName)` when `localDbName == null` and does not also canonicalize the local cache key under `lower_case_database_names = 1`:

https://github.com/apache/doris/blob/54e0e9712b6057d5f1c643103d1612d196b3cd35/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L923-L928

Normal database lookup does canonicalize the local key first:

https://github.com/apache/doris/blob/54e0e9712b6057d5f1c643103d1612d196b3cd35/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L760-L778

https://github.com/apache/doris/blob/54e0e9712b6057d5f1c643103d1612d196b3cd35/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java#L1296-L1303

When an HMS CREATE_DATABASE or ALTER_DATABASE rename event carries a mixed-case remote name while `lower_case_database_names = 1`, the event path may therefore publish a raw-case local database name into the database names snapshot, the database object cache, and the `dbIdToName` navigation map. Later normal mode-1 lookups will canonicalize the lookup name to lowercase first, so the event-published state no longer follows the same local-key convention as ordinary database lookup and reload paths.

This problem is closely related to item 5, but it is not identical:

* item 5 is about generating an unstable database ID from the remote name instead of the canonical local name;
* this item is about publishing the event-created database state under a non-canonical local cache key.

Both should be fixed by resolving the canonical local database name first and then using that same name consistently for event registration, cache publication, invalidation, and ID generation.

## 7. Database/table CREATE, DROP, and RENAME events do not invalidate connector-owned caches

**Status: Open. This is present after connector SPI PR #64304 and is not introduced by #65126.**

After #64304, HMS table metadata, partition metadata, column statistics, derived partition views, and file listings can be owned by the connector. The event driver currently applies database/table REGISTER, UNREGISTER, and RENAME descriptors only through FE CatalogMgr mutations. Those mutations update FE names/object/ID state and engine caches, but they do not call Connector.invalidateTable or Connector.invalidateDb.

REFRESH TABLE, REFRESH DATABASE, direct connector DDL/replay, and partition events already invalidate connector caches. Database/table incremental events are the remaining asymmetric path.

One concrete failure sequence is:

1. Query an HMS table and warm connector-side table/schema caches.
2. Drop the table remotely.
3. Recreate a table with the same database/table name but a different schema.
4. Process the DROP and CREATE events.
5. FE state converges to the recreated object, but the connector may continue serving the old table metadata until TTL or an explicit refresh.

A same-name ALTER VIEW recreate has the same risk because it is represented as an unregister/register table descriptor.

### Suggested fix for item 7

- Invalidate the connector source key before UNREGISTER.
- Invalidate the connector target key before REGISTER.
- Invalidate both source and target for database/table RENAME.
- Use remote names directly from MetastoreChangeDescriptor and do not perform remote lookup or catalog warming.
- Add tests for same-name DROP+CREATE, same-name view recreate, and cross-database rename.

This item may be fixed in a follow-up PR because it is a #64304 master-baseline gap rather than a regression introduced by #65126. Until then, #65126 should describe its rename guarantee as FE metadata-cache convergence rather than complete connector-cache convergence.

Start with CatalogMgr.registerExternalDatabaseFromEvent(), HMSExternalCatalog incremental registration, and ExternalDatabase visibility handling; compare them with ExternalCatalog's canonical naming and full names-loading paths. Trace the remaining CREATE and ALTER event flows and verify that filters, canonical IDs and local keys, and connector-owned cache invalidation remain consistent after events and reloads.

Written by the indexing model from the issue text.