dotCMS / dotCMS/core

Stop Reindex and Make Active discards the reindex queue and promotes a partial index with no confirmation

Open
#37,279 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team : Maintenance
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Summary

The Stop Reindex and Make Active button (Settings → Maintenance → Index) deletes every remaining row from dist_reindex_journal and requests promotion of the partially-built index — with no confirmation step, no indication of how many queued records are about to be discarded, and no warning that the partial index will become the index serving the public site.

The adjacent Stop Reindex Process button is the safe twin, and the two are visually and nominally very similar.

Affected version

26.08.03-01 (code path unchanged on main at time of writing)

Steps to reproduce

  1. On an instance with a substantial content set (~90k contentlets), start a full reindex from Settings → Maintenance → Index.
  2. Wait until progress reaches any partial value (5–30% is enough).
  3. Click Stop Reindex and Make Active.

Expected

A confirmation step that states how many queued records will be discarded and warns that the partially-built index will become active. Ideally the promotion is refused outright (or requires an explicit override) when the candidate index holds materially fewer documents than the currently active one.

Actual

The remaining queue is deleted in a single statement, silently, and the partial index is queued for promotion. Nothing warns the operator, and nothing records how much work was thrown away.

Code

ContentletIndexAPIImpl.stopFullReindexationAndSwitchover():

try {
    ReindexThread.pause();
    queueApi.deleteReindexRecords();   // discards the remaining queue
    this.reindexSwitchover(true);      // requests promotion of the partial index
} finally {
    ReindexThread.unpause();
}

ReindexQueueFactory.deleteReindexRecords():

dc.setSQL("DELETE From dist_reindex_journal where priority >= ? and  priority < ? ");
dc.addParam(Priority.REINDEX.dbValue());   // 300
dc.addParam(Priority.ERROR.dbValue());     // 400

The REST entry point defaults to the promoting variant, so a bare DELETE /api/v1/esindex/reindex promotes:

// ESIndexResource.stopReindexation
@DefaultValue("true") @QueryParam("switch") boolean switchMe
...
if (switchMe) { APILocator.getContentletIndexAPI().stopFullReindexationAndSwitchover(); }
else          { APILocator.getContentletIndexAPI().stopFullReindexation(); }

The non-promoting twin (stopFullReindexation()fullReindexAbort()) clears the reindex slots and leaves the active index alone — the correct behaviour when abandoning a reindex.

Impact

Observed in production. A full reindex was stopped roughly 31% of the way through; ~62,700 queued records were discarded, and a partially-built index containing 27,576 live documents subsequently replaced a complete one containing 88,510. Public pages returned 404s for around two and a half hours until the previous index was reactivated. No content was lost — the database was untouched throughout — but the site was serving from a two-thirds-empty index.

Suggested fix

  • Add a confirmation dialog naming the number of records that will be discarded and stating that the partial index will go live.
  • Add a completeness guard on the switchover: refuse (or require explicit override) when the candidate index document count is materially below the active index.
  • Reconsider switch=true as the API default.

Related

Filed alongside a set of issues covering the same incident chain:

  • #37280 — progress reports 100% complete once the queue is cleared
  • #37281 — a refused forced switchover stays armed and fires later
  • #37282 — index-name timestamp vs elapsed-time parse disagree
  • #37283 — IndiciesInfo.Builder.copy() drops reindexLive (found during the same investigation, unrelated to this chain)

Related Freshdesk ticket: https://helpdesk.dotcms.com/a/tickets/38957

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing ContentletIndexAPIImpl.stopFullReindexationAndSwitchover(), ReindexQueueFactory.deleteReindexRecords(), and ESIndexResource.stopReindexation, then compare them with the non-promoting stopFullReindexation() path. Determine how the maintenance UI and REST default should communicate discarded records and partial-index promotion, and verify that abandoning a reindex cannot silently activate an incomplete index.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.