Full reindex reports success and switches over while discarded journal entries leave content missing from the index

Problem Statement

A full reindex declares itself complete and switches the new index into production even when journal entries were discarded, so the newly activated index is silently missing content. No error is logged at switchover and the admin UI reports the reindex as successful.

The gate that decides "is the reindex done?" and the rule that parks a failed entry disagree about where the queue ends:

Location	Code	Threshold
ReindexThread.switchOverIfNeeded()	inFullReindexation() && queueApi.recordsInQueue() == 0	—
ReindexQueueFactory.recordsInQueue()	select count(*) ... where priority < ? with Priority.ERROR	< 400
ReindexQueueFactory.markAsFailed()	after errorCount >= RETRY_FAILED_INDEX_TIMES (default 5) → priority = Priority.ERROR.dbValue() + priority	>= 400

An entry that exhausts its retries is moved to priority >= 400. recordsInQueue() counts only priority < 400, so parked entries are invisible to the completion check. The queue looks empty, switchOverIfNeeded() fires, and the index carrying the gap becomes the live index.

The same blind spot means the reindex cannot get stuck on these entries either — it is strictly a silent-loss problem, not a hang.

Impact

Any condition that makes an individual contentlet fail to map five times (unreadable binary, corrupt metadata, hung storage I/O, mapping timeout) removes that contentlet from the search index for good, with no operator-visible signal. It is only noticed when someone reports that content is missing from search.

This is pre-existing behavior, not introduced by #37041 — it was found while validating that fix in a cloud environment.

Steps to Reproduce

Discovered on a cloud instance running the branch for #37041, but the mechanism is version-independent.

1. Make per-entry mapping fail deterministically for a subset of content. In the reproduction this was done with an artificially low mapping timeout plus forced metadata regeneration:

   DOT_REINDEX_CONTENTLET_MAPPING_TIMEOUT_SECONDS=1
   DOT_ALWAYS_REGENERATE_METADATA_ON_REINDEX=true
   

   (Note: file size alone is not enough — the reindex reuses metadata generated at upload time and never reads the binary unless regeneration is forced.)
2. Start a full reindex.
3. Let the affected entries fail RETRY_FAILED_INDEX_TIMES (default 5) times. Each failure logs Can't get a mapping for contentlet with id_lang:<id>.
4. Wait for the reindex to finish.

Expected: the reindex either refuses to switch over, or completes with a clear warning naming how many entries were discarded.

Actual: the reindex reports success and switches over. The affected contentlets are absent from the live index. The only trace is the per-entry Can't get a mapping... line, buried among the rest of the reindex logging, with nothing tying it to the switchover decision.

Confirmation query (requires DB access — part of the problem is that there is no other way to see this):

SELECT count(*) FROM dist_reindex_journal WHERE priority >= 400;

Acceptance Criteria

• The full-reindex completion check accounts for discarded entries — switchOverIfNeeded() no longer treats a queue containing priority >= Priority.ERROR rows as empty.
• When discarded entries exist at the moment a full reindex would switch over, the outcome is explicit rather than silent. Decide and document which of these applies:
  • refuse to switch over and leave the current index live, or
  • switch over but log an ERROR naming the discarded count and a sample of affected identifiers.
• The discarded-entry count is observable without DB access — exposed through the reindex/index admin UI, the reindex-mapping health check, or both.
• An operator has a supported way to requeue discarded entries that does not require hand-written SQL.
• Integration test: given a journal with at least one entry at priority >= Priority.ERROR, a full reindex does not silently switch over and report success.
• Integration test: recordsInQueue() (or its replacement in the completion gate) reflects discarded entries.

Open design question for the reviewer: blocking the switchover is the safe default but means one permanently unmappable contentlet can hold an entire reindex hostage. Loud-warning-plus-switchover keeps reindexes unblockable at the cost of requiring someone to read the log. A hybrid — switch over, but surface a persistent warning in the UI until the discarded rows are dealt with — may be the right balance. Worth settling before implementation.

dotCMS Version

Reproduced on a cloud environment running the branch for #37041 (issue-37038-reindex-mapping-permit-leak, based on main). The code paths involved (ReindexThread.switchOverIfNeeded, ReindexQueueFactory.recordsInQueue / markAsFailed) are long-standing and unchanged by that PR, so all currently supported versions are expected to be affected.

Severity

High - Major functionality broken

Links

• Found while validating #37038 / PR #37041
• NA (no Freshdesk ticket)

Start with ReindexThread.switchOverIfNeeded() and ReindexQueueFactory.recordsInQueue() and markAsFailed(), tracing how Priority.ERROR entries are counted and parked. Review the reindex-mapping health check and admin UI entry points mentioned in the acceptance criteria, then add integration coverage for discarded journal entries; done means no silent successful switchover and an operator-visible recovery path.