Failed reindex records are never retried, and clearing them silently abandons unindexed content

Problem Statement

Once a contentlet exhausts its reindex retries it becomes permanently invisible: never retried,
impossible to diagnose accurately, and erased without trace by the only cleanup action available.
Three defects compound into content that is missing from the index with nothing recording it.

1. Failed records are permanently dead

ReindexQueueFactory.markAsFailed() escalates priority past the point the dequeue query can see:

// ReindexQueueFactory.java:270
final int newPriority =
        (idx.errorCount() >= REINDEX_MAX_FAILURE_ATTEMPTS) ? Priority.ERROR.dbValue()
                + idx.getPriority() : (1 + idx.getPriority());

With Priority { ASAP, NORMAL, STRUCTURE, REINDEX, ERROR } and dbValue() = ordinal * 100, a record
starting at REINDEX (300) reaches 305 after the 5 default retries
(RETRY_FAILED_INDEX_TIMES), then jumps to 400 + 305 = 705.

The dequeue query only picks up priority <= ERROR(400):

// ReindexQueueFactory.java:336
"select * from dist_reindex_journal where MOD(id, ?) = ? and priority <= ? and id > ?
   ORDER BY priority ASC LIMIT 2000"
// addParam(Priority.ERROR.dbValue())  ->  400

705 > 400, so the record is never considered again. Nothing in the system will ever retry it, even
after the underlying cause is fixed. Observed: 380 records parked at priority 705, unchanged across
a dotCMS restart and 15 hours of uptime.

2. Clearing failed records discards the content, not just the rows

deleteFailedRecords() is a bare delete with no re-enqueue:

// ReindexQueueFactory.java:180
protected void deleteFailedRecords() throws DotDataException {
    DotConnect dc = new DotConnect();
    dc.setSQL("DELETE From dist_reindex_journal where priority > ?");
    dc.addParam(Priority.REINDEX.dbValue());
    dc.loadResult();
}

Exposed as DELETE /api/v1/esindex/failed and as the UI's clear action. Because those contentlets
were never indexed, deleting the rows removes the only record that they are missing. The content
is silently absent from the index afterwards, and nothing will surface it.

In the case that prompted this report, 370 contentlets were absent from the live Elasticsearch
indices registered in the indicies table. They survived only in the previous reindex generation
(*_20260826165345), which is unregistered and scheduled for deletion by DeleteOldESIndicesJob.
Clearing the failed queue at that point would have made the loss unrecoverable and undetectable.

3. failureReason reports the batch error, not the document's

Because a bulk failure discards the whole batch (see the companion issue on the ES bulk limit),
every record in that batch is stamped with the same cause. All 380 records reported the identical
byte count:

String value length (20054016) exceeds the maximum allowed (20000000, ...)

including contentlets under 100 KB such as "Zeste" and "9". Of the 380, only 10 were actually
oversized. The remaining 370 are indistinguishable from the real culprits in both the API response
and the UI, so the field actively misleads diagnosis — it points at the wrong content.

4. GET /api/v1/esindex/failed is unusable at the moment it is needed

downloadRemainingRecordsAsCsv embeds the full contentlet map per record:

// ESIndexResource.java:195
Map<String, Object> conMap = Try.of(() -> new ContentletToMapTransformer(contentlet).toMaps().get(0))
        ...
failure.put("contentlet", conMap);

With 380 records whose failure cause was oversized bodies, the response was 176 MB — it embeds
exactly the 16 MB documents that caused the problem. A diagnostic endpoint becomes impractical
precisely when the failures are large-document failures. (The method name also says CSV while it
returns JSON.)

Steps to Reproduce

1. Cause reindex failures that exhaust the retry budget — e.g. one contentlet with a body
   over 20 MB batched with several hundred healthy ones (see the companion issue), or any
   reproducible indexing error.

2. Run a full reindex and wait for the retries to be exhausted.

3. Dead records: confirm the rows are parked above the dequeue threshold and are never
   retried, even after fixing the root cause:

   SELECT priority, count(*) FROM dist_reindex_journal GROUP BY priority;   -- 705
   

   Restart dotCMS and re-run a reindex — the count does not move.

4. Misleading cause: compare index_val against the real document size:

   SELECT j.ident_to_index,
          pg_size_pretty(length(c.contentlet_as_json::text)::bigint),
          left(j.index_val, 80)
   FROM dist_reindex_journal j
   JOIN contentlet c ON c.identifier = j.ident_to_index
   ORDER BY length(c.contentlet_as_json::text) DESC;
   

   Every row carries the same message; only a handful are actually oversized.

5. Oversized response: GET /api/v1/esindex/failed and measure the payload.

6. Silent abandonment: verify the affected contentlets are absent from the index
   registered in indicies, call DELETE /api/v1/esindex/failed, and observe that they
   remain absent with no remaining record anywhere.

Acceptance Criteria

• Failed records remain reachable for retry, or there is an explicit, discoverable way to
  requeue them (an API/UI "retry failed" that re-enqueues at a processable priority).
• Clearing failed records either re-enqueues the affected contentlets or warns
  explicitly that they are not in the index and will be dropped from tracking.
• failureReason distinguishes the document that caused the failure from documents that
  merely shared its batch.
• GET /api/v1/esindex/failed returns a bounded summary by default (identifier, inode,
  title, cause, size); full contentlet payloads are opt-in and/or paginated.
• downloadRemainingRecordsAsCsv is renamed, or actually returns CSV.

dotCMS Version

dotcms/dotcms:trunk. PostgreSQL 16, ~1.55 M contentlets, OpenSearch 1.3.20 + 3.4.0 in
PHASE_1_DUAL_WRITE_ES_READS.

Severity

High - Major functionality broken

High — content silently missing from the search index, and the documented cleanup path destroys
the evidence.

Links

• ReindexQueueFactory.java:270 — markAsFailed, priority escalation past the dequeue threshold
• ReindexQueueFactory.java:336 — dequeue query, priority <= ERROR(400)
• ReindexQueueFactory.java:180 — deleteFailedRecords, delete without re-enqueue
• ESIndexResource.java:180 — downloadRemainingRecordsAsCsv

Freshdesk ticket: NA — found during internal ES→OpenSearch migration testing.