A client-side bulk failure discards the entire batch — inherited by OpenSearch in Phase 3

Problem Statement

When a bulk index request fails on the client side — before the server ever sees it — every
document in the batch is marked failed, not just the one that caused it. With the tuning-guide
setting DOT_REINDEX_THREAD_ELASTICSEARCH_BULK_ACTIONS=1000, one bad document takes up to 999
healthy ones with it.

Measured on a real customer dataset (1.55 M contentlets, full reindex):

Failed contentlets by size	Count
> 10 MB — the actual cause	10
1–10 MB	4
100 KB – 1 MB	3
< 100 KB — pure collateral damage	363

10 oversized documents produced 380 failures. 363 of them were under 100 KB and had nothing
wrong with them.

Both engine paths have the same structure

Each path isolates failures the server reports per item, and discards the whole batch on a
client-side exception.

Elasticsearch — ContentletIndexOperationsES.java:239 (per-item, isolated) vs :261 (whole batch):

public void afterBulk(final long executionId, final BulkRequest request, final Throwable failure) {
    listener.afterBulk(executionId, failure);        // ES: whole batch
}

OpenSearch — ContentletIndexOperationsOS.java:157 flush():

try {
    final BulkResponse response = client.bulk(BulkRequest.of(b -> b.operations(batch)));
    for (final BulkResponseItem item : response.items()) {   // per-item, isolated
        ...
    }
    listener.afterBulk(executionId, results);
} catch (final Exception e) {
    listener.afterBulk(executionId, e);              // OS: whole batch  (:186-187)
}

Both converge on BulkProcessorListener.afterBulk(long, Throwable) — BulkProcessorListener.java:171:

public void afterBulk(final long executionId, final Throwable failure) {
    final String msg = failure != null ? failure.getMessage() : "(no message)";
    if (shadow) {
        logShadowWriteFailure(this.getClass(),
                "[OS] Bulk process failed entirely (fire-and-forget): " + msg, failure);
        return;                                                    // logged, nothing recorded
    }
    Logger.error(ReindexThread.class, "Bulk process failed entirely: " + msg, failure);
    workingRecords.values().forEach(idx -> handleFailure(idx, msg));   // every record in the batch
}

The ES→OS migration does not resolve this — Phase 3 inherits it

The trigger described below happens to be Elasticsearch-specific, so it is tempting to treat the
whole defect as something the migration retires. It is not. The primary listener is constructed
with shadow = false regardless of engine (BulkProcessorListener.java:69):

BulkProcessorListener() {
    this(IndexConfigHelper.MigrationPhase.current().isMigrationComplete()
            ? IndexTag.OS : IndexTag.ES, false);
}

and the shadow factory's own javadoc states the consequence:

In Phase 3, OS becomes the primary and this factory is no longer used — the caller passes the
standard BulkProcessorListener directly.

So in Phase 3 the OpenSearch path runs with shadow = false and executes
workingRecords.values().forEach(idx -> handleFailure(idx, msg)) — the exact behaviour ES has
today. OpenSearch also has its own client-side ceiling (http.max_content_length, 100 MB by
default); a large enough batch crosses it and lands in the same catch. Same failure mode,
different threshold.

A related gap: in dual-write phases, OS bulk failures are recorded nowhere

The if (shadow) { …; return; } branch above means that during Phases 1 and 2 an OpenSearch bulk
failure produces a log line at DOTCMS_SHADOW_WRITE_LOG_LEVEL (default WARN) and nothing
else — no dist_reindex_journal entry, no errorCount, no UI signal. Fire-and-forget is the
deliberate design for a shadow store, but it means OS-side divergence accumulates with no record
an operator can query. (See the companion issue on per-engine failure visibility.)

---

The trigger that exposed it: a 20 MB Jackson limit with no way to raise it

A contentlet whose generated document contains a single string value larger than 20,000,000
characters throws during serialization, inside the Elasticsearch client — not in any dotCMS
ObjectMapper:

com.fasterxml.jackson.core.exc.StreamConstraintsException: String value length (20054016)
  exceeds the maximum allowed (20000000, from `StreamReadConstraints.getMaxStringLength()`)
    at com.fasterxml.jackson.core.JsonGenerator.copyCurrentStructure(JsonGenerator.java:2638)
    at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.copyCurrentStructure(JsonXContentGenerator.java:405)
    at org.elasticsearch.common.xcontent.XContentBuilder.copyCurrentStructure(XContentBuilder.java:1003)
    at org.elasticsearch.client.RequestConverters.bulk(RequestConverters.java:243)
    at org.elasticsearch.client.RestHighLevelClient.bulkAsync(RestHighLevelClient.java:549)
    at com.dotcms.content.elasticsearch.business.ContentletIndexOperationsES.lambda$createBulkProcessor$0(ContentletIndexOperationsES.java:270)
    at com.dotmarketing.common.reindex.ReindexThread.runReindexLoop(ReindexThread.java:238)

dotCMS writes the document fine (writeValueAsString — writing has no constraint). The failure is
on the re-parse that RequestConverters.bulk() performs to copy the document into the bulk
request body, using ES's own XContentBuilder / JsonFactory.

The limit is not configurable on this path. StreamReadConstraints is configured in exactly
one place in the entire repository — ContentletJsonHelper.java:43
(CONTENTLET_JSON_MAX_STRING_LENGTH_MB, default 100 MB):

final int maxStringLengthMb = Config.getIntProperty("CONTENTLET_JSON_MAX_STRING_LENGTH_MB", 100);
objectMapper.getFactory().setStreamReadConstraints(
        StreamReadConstraints.builder().maxStringLength(maxStringLengthMb * 1024 * 1024).build());

The ES client path never sees it, so raising that property has no effect. Every other mapper —
including ESMappingAPIImpl.createMapper() — uses the bare Jackson default. This arrived with the
Jackson 2.15+ upgrade (currently 2.17.2 per bom/application/pom.xml), which introduced the
20 MB default maxStringLength; the ES 7.x XContentBuilder predates the setting and offers no
way to configure it. StreamReadConstraints.overrideDefaultStreamReadConstraints() — a static
global that applies to every JsonFactory created afterwards, including ES's — is present in the
jackson-core-2.17.2.jar currently shipped.

The reported size is a buffer boundary, not the document's size

The number in the message is not the offending value's length, which makes it useless for finding
the culprit. Two contentlets with very different bodies — one ~16 MB, one ~9.9 MB — both report
the byte count 20054016, identical to the digit. That is impossible if it were the actual
string length. StreamReadConstraints.validateStringLength() is called from
TextBuffer.finishCurrentSegment(), so the value reported is the buffer's accumulated segment
size when the check trips, not the value being parsed.

It also means the effective per-document budget is smaller than 20 MB. The body appears more than
once in the generated document (<type>.body, <type>.body_dotraw, and again inside catchall),
so accumulation crosses the threshold at roughly half the nominal limit. Measured boundary:

Group	Largest contentlet_as_json
Failed the limit	10,138 kB (9.90 MiB)
Contained base64 but passed (1,856 identifiers)	9,527 kB (9.30 MiB)

A body around 9.5 MB already fails against a nominal limit of 20 MB. Anyone sizing content
against the error message will conclude they have twice the headroom they actually have.

The ES path is affected; the OpenSearch shadow-write path is not

During the same full reindex in PHASE_1_DUAL_WRITE_ES_READS, the 370 collateral contentlets
landed in OpenSearch 3 but never in Elasticsearch:

New index	Docs present (of the 370 affected identifiers)
cluster_<id>.working_<ts> (ES path)	0
cluster_<id>.working_<ts>.os (OS shadow)	524
cluster_<id>.live_<ts> (ES path)	0
cluster_<id>.live_<ts>.os (OS shadow)	406

(Counts exceed 370 because of multiple language versions per identifier.)

100 % success on the OpenSearch side, 100 % failure on the Elasticsearch side. Reads in Phase 1 are
served from ES, so those contentlets were unsearchable while being perfectly present in OpenSearch.
Total document counts concealed it: the two working indices differed by only 45 documents overall
while ES was missing 524 of these and holding ~569 others OpenSearch lacked.

How the oversized content is actually authored

Worth recording, because it determines where a guard can live. The production content was not
created by pasting screenshots — it was pasted from a Google Doc. The stored markup still carries
the origin marker:

<b id="docs-internal-guid-a20ecb2f-7fff-f647-18ce-3073767e2643">
  <img height="132" src="data:image/png;base64,…" title="5.png" width="132" />
  <img height="133" src="data:image/png;base64,…" title="6.png" width="133" />
</b>

Embedded image	base64 payload	decoded PNG	rendered at
5.png	7,287 kB	5,465 kB	132×132 px
6.png	7,486 kB	5,615 kB	133×133 px
(untitled)	1,356 kB	1,017 kB	full size

Two ~5.5 MB images displayed as 132-pixel thumbnails. The author had no visual cue that 11 MB had
just been embedded. Stripping only the base64 payloads takes the same document from 16,517,678
characters to 1,914 — a 99.99 % reduction with no loss of authored content.

No editor-level setting mitigates this. TinyMCE's paste_data_images governs images pasted
from the clipboard as files; here the data URIs arrive inside pasted text/HTML, which every
field type accepts. The Block Editor's uploadAsset() path (asset-uploader.extension.ts:190)
only intercepts clipboardData.files for the same reason, and image.node.ts:34 sets
allowBase64: true with a parse rule of img[src]. A guard therefore has to be server-side.

Scale on the affected install: 1,860 identifiers / 8,405 versions / 12 GB of embedded base64,
of which only the handful above ~9.5 MB actually break indexing. The rest is silent bloat carried
through every DB read, cache entry and index document.

Steps to Reproduce

1. Create a content type with an indexed text field. All three field types reproduce —
   WysiwygField, TextAreaField and StoryBlockField. Verified on a content type carrying
   one of each: the same payload landed in all three, 10 data URIs and ~1.6 MB per field.

2. Give one contentlet a field value larger than ~9.5 MB. Any of these work:

   a. Paste from Google Docs / Word — the production vector. Put two or three multi-MB
   images in a Google Doc, resize them small, select all, copy, paste into the field. Google
   Docs places the images in the clipboard as data:image/png;base64,… inside the pasted
   HTML, and the editor stores them verbatim.

   b. Paste text containing data URIs. Markdown or plain text with
   ![](data:image/png;base64,…) reproduces it in every field type, including
   TextAreaField — a plain <textarea> accepts a data URI written out as text.

   c. Directly, for a deterministic size:

   UPDATE contentlet
   SET contentlet_as_json = jsonb_set(contentlet_as_json, '{fields,body,value}',
         to_jsonb('<p>' || repeat('A', 21*1024*1024) || '</p>'))
   WHERE inode = '<INODE>';
   

   What does not reproduce it: dragging and dropping an image (uploaded as a dotAsset —
   the correct behaviour), and pasting from GitHub comments (images arrive as https:// links).

3. Queue that contentlet together with several hundred healthy ones in the same batch.

4. Set DOT_REINDEX_THREAD_ELASTICSEARCH_BULK_ACTIONS=1000 and run a full reindex.

5. Observe Bulk process failed entirely in the log, and the whole batch — not just the
   oversized document — sitting in dist_reindex_journal.

6. Set CONTENTLET_JSON_MAX_STRING_LENGTH_MB=500 and repeat: the failure is unchanged.

7. Compare BULK_ACTIONS=10 vs 1000 to see the blast radius scale with batch size.

Acceptance Criteria

• A document that cannot be serialized fails individually; the remaining documents in the
  batch are indexed. This applies to both ContentletIndexOperationsES and
  ContentletIndexOperationsOS — the OpenSearch path inherits the current behaviour in
  Phase 3, so fixing only the ES side leaves the defect in place after the migration.
• The failure is attributed to the offending contentlet (identifier + inode + field + size),
  not to every record that shared its batch.
• The 20 MB read limit is configurable for the ES bulk path, or raised to match the existing
  CONTENTLET_JSON_MAX_STRING_LENGTH_MB intent, via
  StreamReadConstraints.overrideDefaultStreamReadConstraints() at startup.
• The size reported in the failure message is the offending value's length, not the parser's
  accumulated buffer size.
• In dual-write phases, an OpenSearch bulk failure leaves a queryable record rather than only
  a WARN line, so shadow-side divergence is detectable.
• Content that cannot be indexed is prevented at the point of authoring, not only at the point
  of indexing. Since the payload arrives as text, this has to be a server-side guard on save
  (size ceiling and/or data-URI rejection for indexed fields); no editor setting covers
  WysiwygField, TextAreaField and StoryBlockField at once.

dotCMS Version

dotcms/dotcms:trunk, jackson.version 2.17.2, opensearch.version 3.3.0.
Reproduced against OpenSearch 1.3.20 (ES-compatible endpoint) + OpenSearch 3.4.0 in
PHASE_1_DUAL_WRITE_ES_READS, PostgreSQL 16, ~1.55 M contentlets.

Severity

High - Major functionality broken

Silent, size-dependent content loss from the search index. In dual-write phases it also produces
undetected divergence between the two engines, and the batch-discard behaviour is inherited by
OpenSearch when it becomes primary in Phase 3.

Links

• BulkProcessorListener.java:171 — afterBulk(long, Throwable), marks the whole batch failed
• BulkProcessorListener.java:69 — primary listener always constructed with shadow = false
• ContentletIndexOperationsOS.java:186 — OS catch (Exception e) → whole batch
• ContentletIndexOperationsES.java:261 — ES equivalent
• ContentletJsonHelper.java:43 — the only place StreamReadConstraints is configured
• ESMappingAPIImpl.java:161 — createMapper(), bare new ObjectMapper()
• ContentletIndexOperationsES.java:270 — bulk processor entry point
• ReindexThread.java:238 — runReindexLoop

Freshdesk ticket: NA — found during internal ES→OpenSearch migration testing.