One undeserialisable payload destroys delivery of every event in its poll window

Problem Statement

SystemEventsAPI.getEventsSince(...) converts a polled batch of system_event rows as a whole. If any single row's payload cannot be deserialized, the entire batch throws — and every other event in that poll's window is lost, not just the bad one.

The mechanism

SystemEventsFactory$SystemEventsAPIImpl.getEventsSince (SystemEventsFactory.java:295) hands the whole result set to ConversionUtils.convert, which calls convertSystemEventDTO (SystemEventsFactory.java:395) per row. That unmarshals the stored payload through JacksonMarshalUtilsImpl. Payload records the payload's concrete class name (Payload.java:88 — this.type = rawData.getClass().getName()), and the receiving node reconstructs the data into that class.

So one payload class that Jackson cannot construct is enough to fail the conversion of the batch. What happens next makes it worse rather than better:

1. getEventsSince throws.
2. AbstractJobDelegate.execute (AbstractJobDelegate.java:33-39) catches and logs a single ERROR line, without propagating.
3. SystemEventsJob treats the poll as finished and advances its cursor past the whole window.

The result is bulk event loss on the cluster-wide invalidation path, announced by one log line that names a Jackson error rather than the events that were dropped.

A concrete instance (already fixed separately)

com.dotcms.business.SystemTableUpdatedKeyEvent had a single unannotated constructor and no default constructor, so it could be written but never read back. It is the payload of the CLUSTER_WIDE_EVENT published by SystemTableImpl.delete() — meaning that event could never be consumed by another node, and its presence in a window also destroyed delivery of every other event there. That class has been given an explicit @JsonCreator in the fix for #36827.

Any other payload class with the same shape has the same effect today, and nothing prevents a new one from being introduced. The per-class fix does not address the batch-level fragility.

Steps to Reproduce

1. On a two-node cluster (or simulated by writing a row with a foreign server_id), publish a CLUSTER_WIDE_EVENT whose payload data is an object Jackson can serialize but not deserialize — i.e. a class with a single unannotated constructor and no default constructor.
2. Publish one or more ordinary, well-formed events with created timestamps in the same poll window.
3. Let another node's SystemEventsJob poll that window.

Expected: the malformed row is logged and skipped; the well-formed events in the same window are delivered normally.

Actual: getEventsSince throws for the whole batch, a single ERROR line is logged by AbstractJobDelegate, no events from that window are delivered, and the cursor advances past all of them.

Acceptance Criteria

• A single undeserialisable payload no longer prevents delivery of the other events in the same batch — rows are converted individually.
• A row that cannot be converted is logged with enough detail to identify it (event identifier, event type, payload class) and skipped, rather than failing its batch.
• Skipped-undeserialisable events are countable — a warning and/or metric, so the condition is visible in aggregate rather than as isolated stack traces.
• Automated test: a batch containing one undeserialisable payload plus several well-formed events delivers all the well-formed ones.
• Automated test: the undeserialisable event is reported (log/metric), not silently discarded.
• No behaviour change when every payload in a batch is well-formed.

dotCMS Version

Present on main. Not version-specific — the batch-conversion structure has been in place since the system event queue was introduced.

Severity

High - Major functionality broken

Links

• Discovered while implementing the fix for #36827 (system events silently dropped in a cluster); the SystemTableUpdatedKeyEvent instance is fixed there, this issue covers the underlying batch fragility.
• NA (no Freshdesk ticket of its own; related to Freshdesk #38560 via #36827)

Start in SystemEventsFactory.java at getEventsSince, convertSystemEventDTO, and the ConversionUtils path; then trace error handling through AbstractJobDelegate.java and SystemEventsJob. Reproduce a batch with one undeserialisable payload and several valid events. Done means valid events are delivered, the bad row is identifiable in logs or metrics, and automated coverage confirms both outcomes.