[Bug][Meta Cache] Complete global invalidation ordering after PR #65126
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Background
PR #65126 now fixes the per-key ordering problem between object cache entries and ID-to-name navigation state for external metadata caches.
The current PR covers the key-scoped paths below for both database and table layers:
- hot named lookup
- miss-load / generation-rejected load
- register / update
- invalidate / unregister / rename
So this issue no longer tracks those per-key races.
### Remaining gap
The remaining correctness gap is on full invalidation / reset style paths that still update object caches and ID maps in separate steps, without a full publication barrier across all affected keys / stripes.
Examples include patterns like:
- `tables.invalidateAll(); tableIdToName.clear();`
- `databases.invalidateAll(); dbIdToName.clear();`
With the current structure, a concurrent named lookup / load / publication may still race with `invalidateAll()` / full reset and repopulate object or ID state after cleanup has started.
### Why this is a follow-up issue
Fixing this correctly is broader than the per-key ordering work already merged into PR #65126.
A proper solution likely needs:
- a full invalidation barrier or equivalent global generation protocol
- fixed-order coordination across stripes / keys to avoid deadlock
- careful interaction with manual load, refresh, and removal-listener paths
- dedicated deterministic concurrency tests
That is a larger change than the current PR scope.
### Expected scope
This follow-up should focus on the remaining non-per-key cases, including:
1. full invalidation / reset paths for table caches
2. full invalidation / reset paths for database caches
3. interactions between invalidate-all style cleanup and concurrent named lookup / miss-load / publication
### Expected guarantees
At a minimum, the final fix should ensure:
- `invalidateAll()` / full reset cannot be undone by a concurrent late publication
- object cache and ID navigation state are cleared under a consistent global ordering rule
- the per-key guarantees already added in PR #65126 remain unchanged
- replay stays cache-only
- no remote I/O is introduced under publication locks
### Test expectations
Please add deterministic concurrency tests for at least:
- `invalidateAll()` / full reset vs hot named lookup
- `invalidateAll()` / full reset vs blocked miss-load
- both table and database cache paths
### Related context
- PR #65126
- Review discussion: https://github.com/apache/doris/pull/65126#discussion_r3642519993
Contributor guide
Research direction
Start by tracing the table and database cache paths around invalidateAll(), clear(), named lookup, miss-load, and publication, using PR #65126 and its review discussion for existing ordering rules. Add deterministic concurrency tests for full invalidation versus hot lookup and blocked miss-load on both cache paths. Done means late publication cannot undo cleanup, object and ID state follow one global ordering, replay remains cache-only, and publication locks perform no remote I/O.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100