Site Copy: decouple scroll read from per-item write to remove keep-alive coupling on large sites

Description

Summary

Site Copy (HostAssetsJobImpl) consumes the Elasticsearch/OpenSearch Scroll API lazily while doing expensive per-item writes inside the scroll loop. Because the scroll's server-side context is held open across that per-item work, the fixed ES_SCROLL_KEEP_ALIVE_MINUTES (default 5 min) becomes coupled to how long the consumer takes to process one batch. On large sites the per-batch processing time exceeds the keep-alive, the scroll context expires, and the next continuation fails with search_phase_execution_exception / all shards failed — aborting the copy partway through.

The mechanism

1. PaginatedContentlets switches to the Scroll API when a result set exceeds ES_SCROLL_API_THRESHOLD (default 10,000).
2. The scroll fetches CONTENTLET_PER_PAGE (default 1,000) ids per batch. Keep-alive is 5 min, reset on each fetch (ESContentletScrollImpl lines 41–42, 92, 139).
3. Site Copy iterates the results and, for every item, hydrates + creates + saves + reindexes a new contentlet — inside the same loop that drives the scroll (HostAssetsJobImpl.copySiteAssets).
4. So the wall-clock between two scroll continuation requests = time to process one full 1,000-item batch. When that exceeds 5 min, OpenSearch has already dropped the context → the next _search/scroll returns "all shards failed".

Evidence (FD #38681 cluster logs)

Signal	Observation
Per-batch processing time (node 1)	Content phase 09:41:41 → 09:48:28 = 6m47s to process the first ~1,000-item batch — exceeds the 5-min keep-alive
Failure point	On the first scroll continuation (PaginatedContentlets$ContentletIterator.Error continuing scroll API), ~1,005 items in
Corroborating tell	The clearScroll DELETE /_search/scroll returns HTTP 404 Not Found — the context was already gone, exactly what an expired cursor looks like (a transient shard blip on a live context would not 404)
Concurrent load	~4,047 addIdentifierReindex + ~1,516 index ops during the window — the copy hammers the same cluster it is scrolling, further slowing each batch
Determinism	Fails every attempt on this site; both cluster nodes hit the identical error

Why this improvement is recommended (justification)

This is worth doing as a design change rather than living on config tuning, for five reasons:

1. Config only moves the wall — it never removes it. ES_SCROLL_KEEP_ALIVE_MINUTES=30 unblocks a 77k site, but a site 5× larger, a slower/busier cluster, or a heavier content type walks straight back into the same failure. The timeout is coupled to consumer speed; no single value is correct for all sites.
2. Silent, partial, data-shaped failure. The copy doesn't just error — it produces a half-copied site (in FD #38681, ~1.3% of content copied, sorted alphabetically, so whole classes of files like .vtl/.css/.js simply never get reached). A partially-populated site looks "created" but is broken, which is worse than a clean failure and generates follow-up support load.
3. It holds an OpenSearch resource open for no reason. Keeping a scroll context alive across minutes of writes pins search contexts/segments on a shared cluster (canada-es-1.dotcms.cloud), adding pressure precisely while the copy's own reindex load is peaking. Bigger keep-alive = more resource held = more strain — the mitigation actively works against cluster health.
4. The fix is cheap and low-risk. The scroll already yields lightweight ids, not hydrated objects, and the job already retains O(N) mapping state — so decoupling read from write is a small, contained change with a large blast-radius reduction (see Additional Context for the memory analysis).
5. Recurring, customer-visible surface. Site Copy on large sites is a common enterprise operation; this failure is deterministic and will recur for any customer above the threshold. Fixing the pattern removes a class of tickets, not a single incident.

Net: decoupling the read from the write makes the keep-alive irrelevant to consumer speed — the read completes in seconds, so even the original 1-minute value would be safe — and eliminates the failure mode instead of postponing it.

Proposed approach (design decision to confirm in refinement)

Recommended — decouple read from write: run the scroll once, read-only and fast, to materialize only the id/inode list (tight loop, no per-item work → keep-alive is a non-issue), then run the copy from that list, hydrating one contentlet at a time inside the write loop as today. Smallest conceptual change; removes the coupling entirely.

Acceptance Criteria

• Site Copy of a site with >100,000 contentlets completes successfully without any Error continuing scroll API … all shards failed in the logs.
• The Site Copy content phase no longer holds an open scroll/PIT context across per-item copy work — the read that enumerates source content completes before (or independently of) the per-item write loop.
• With ES_SCROLL_KEEP_ALIVE_MINUTES left at its default, a large-site copy that previously failed now succeeds (i.e. success no longer depends on raising the keep-alive).
• Memory stays bounded during the copy: only lightweight ids (not hydrated Contentlet objects) are materialized up front; full contentlets are hydrated one-at-a-time in the write loop. Verified on a >100k-item site without OOM/heap regression.
• Copied site is complete — all non-page assets (.vtl, .css, .js, file-asset contentlets) that fall alphabetically after the previous failure point are present.
• The clearScroll/context-cleanup path no longer logs the Unable to parse response body error on a 404 (either fixed or made non-applicable by the new approach).
• Behavior is regression-tested for small sites (below ES_SCROLL_API_THRESHOLD) and for the scroll/large-site path.
• A note is added to the copy/scroll config docs describing the new behavior and what ES_SCROLL_KEEP_ALIVE_MINUTES / CONTENTLET_PER_PAGE now do (and don't) affect.

dotCMS Version

Evergreen
26.08.03-01

Priority

Medium

Links

Support ticket : https://helpdesk.dotcms.com/a/tickets/38681

Additional Context

Memory analysis — why "read all ids first" is safe (and where it stops being safe).

The scroll yields ContentletSearch (three strings: inode/identifier/index + a float), not hydrated contentlets, and PaginatedContentlets already reduces each page to a List<String> of inodes; the full Contentlet is hydrated lazily one at a time via contentletAPI.find(inode, …). Materializing the full id list up front therefore costs ~88 bytes/id:

Site size	Heap for the id list
77k (this ticket)	~6.8 MB
100k	~8.8 MB
1M	~88 MB
10M	~880 MB ← back with a temp table / disk-backed queue beyond here

The job already retains O(N) mapping state for the whole run (copiedContentsBySourceId, copiedFoldersBySourceId, copiedContentTypesBySourceId, copiedRelationshipsBySourceId, failedContents, htmlPageInodes), so a List<String> of ids adds no new order of magnitude. The one rule: materialize ids, never hydrated Contentlet objects (a hydrated contentlet is KBs–tens of KBs → 100k × ~10KB ≈ 1 GB → OOM). For multi-million-item sites, back the id list with a temp/tracking table rather than heap — worth naming as the known ceiling so the fix isn't mistaken for infinitely scalable.

Operational mitigation (already available, not a substitute for the fix):

• ES_SCROLL_KEEP_ALIVE_MINUTES=30 (default 5) — extend the context lifetime enough to cover one inter-batch gap.
• CONTENTLET_PER_PAGE=250 (default 1000) — smaller batches → shorter inter-batch gap.
• ES_INDEX_OPERATIONS_TIMEOUT (15s) is not the relevant knob — it applies to the initial search query, which succeeds; the failure is on scroll continuation.

Relevant code:

• dotCMS/src/main/java/com/dotcms/content/elasticsearch/business/ESContentletScrollImpl.java (keep-alive 41–42, 92, 139)
• dotCMS/src/main/java/com/dotmarketing/util/contentet/pagination/PaginatedContentlets.java (threshold 50–51; scroll vs offset 155–174; iterator hydration 285–288)
• dotCMS/src/enterprise/java/com/dotcms/enterprise/priv/HostAssetsJobImpl.java (content-copy phase; O(N) maps 306–313)
• ESIndexAPI.INDEX_OPERATIONS_TIMEOUT_IN_MS (104–105)

History / provenance: introduced by PR #34044 (Closes #33661); scroll code later migrated to OpenSearch under task #34647 (PRs #34691, #34742).

Start with HostAssetsJobImpl.copySiteAssets and trace how PaginatedContentlets drives ESContentletScrollImpl during the content-copy phase. Review the existing pagination and Site Copy regression coverage before changing behavior. Done means large and small copies complete with bounded id-only memory, no scroll continuation failure, and the documented configuration behavior is covered.