dotCMS / dotCMS/core

PushPublisher.process() throws NullPointerException on retry when environment is "Push to one" with multiple endpoints

Open
#35,703 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

OKR : Customer Support stale Team : Maintenance
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Problem Statement

In an environment configured with "Push to one" mode (environment.push_to_all = false) that has more than one enabled, dynamic (http/https) endpoint, any push-publish failure that causes the bundle to be re-queued for a retry will throw a NullPointerException at com.dotcms.publisher.pusher.PushPublisher.process(PushPublisher.java:205):

java.lang.NullPointerException: Cannot invoke "com.dotcms.publisher.business.EndpointDetail.getStatus()" because "epDetail" is null
at com.dotcms.publisher.pusher.PushPublisher.process(PushPublisher.java:205)
at com.dotcms.publishing.PublisherAPIImpl.publish(PublisherAPIImpl.java:163)
at com.dotcms.publisher.business.PublisherQueueJob.execute(PublisherQueueJob.java:201)

Impact:
The original failure reason for the bundle is lost from the UI. The Publishing Queue's per-endpoint "Additional Information" column gets overwritten by the NPE message, so the customer can no longer see why the bundle actually failed in the first place. Debuggability of any push-publish failure in this topology is broken.
The bundle is correctly marked FAILED_TO_SEND_TO_ALL_GROUPS [4], but the audit history can no longer be updated for subsequent attempts (Bundle … is stalled and its Audit History will NOT be updated properly), and the retry mechanism is effectively dead from that point on.

Root cause (from source, dotCMS/src/main/java/com/dotcms/publisher/pusher/PushPublisher.java lines 191–218):

Map<String, EndpointDetail> endpointsDetail = endpointsMap.get(environment.getId());
if (null != allEndpoints) {
for (PublishingEndPoint ep : allEndpoints) {
if (ep.isEnabled() && getProtocols().contains(ep.getProtocol())) {
if (null == endpointsDetail || endpointsDetail.isEmpty()) {
endpoints.add(ep);
} else {
EndpointDetail epDetail = endpointsDetail.get(ep.getId());   // can be null
if (DeliveryStrategy.ALL_ENDPOINTS.equals(this.config.getDeliveryStrategy())
|| (DeliveryStrategy.FAILED_ENDPOINTS.equals(this.config.getDeliveryStrategy())
&& PublishAuditStatus.Status.SUCCESS.getCode() != epDetail.getStatus()   // NPE
&& Status.SUCCESS_WITH_WARNINGS.getCode() != epDetail.getStatus()
&& PublishAuditStatus.Status.BUNDLE_SENT_SUCCESSFULLY.getCode() != epDetail.getStatus())) {
endpoints.add(ep);
}
}
}
}
}
boolean failedEnvironment = false;
if (!environment.getPushToAll()) {
Collections.shuffle(endpoints);
if (!endpoints.isEmpty())
endpoints = endpoints.subList(0, 1);
}

The "first attempt?" guard at line 197 is per-environment, not per-endpoint. Combined with the subList(0, 1) at line 218 (only one endpoint is actually attempted per pass when pushToAll == false), the audit map ends up populated for the one endpoint that was tried, but null for the others. On retry, the FAILED_ENDPOINTS branch dereferences epDetail.getStatus() for those null-detail endpoints and throws.

Steps to Reproduce

Setup (reproduces the exact customer topology — one environment, multiple endpoints, push-to-one mode, post-receipt failure):

https://drive.google.com/file/d/1KlXwSwR-O-WzemBH1yMmHvsZ399cp2WR/view?usp=sharing

  • Spin up one sender dotCMS instance (any 26.03.x build) and two receiver instances on the same network.
  • On the sender, go to System → Push Publishing → Environments and click + Add Environment.
  • In the environment dialog, select the "Push to one" radio button (NOT "Push to all"). Save.
  • Open the new environment and add two enabled endpoints, one pointing at each receiver. Both must be dynamic (http or https) and have an auth key configured that will pass authentication on the receiver (otherwise the bundle takes a fast-fail path that does not reach this bug).
  • On one of the two receivers, edit any content type and add a mandatory, unique field. Leave the other receiver's content type unchanged.

Trigger the failure and retry:

  • On the sender, create a contentlet of that content type and leave the new mandatory field empty (or set it to a value that collides with the unique constraint).
  • Push-publish that contentlet to the environment created in step 3.
  • Push the same bundle 6–10 times in a row. Push-to-one selects one of the two endpoints randomly per attempt; you need at least one push where the randomly-selected receiver is the one with the mandatory-field constraint so that processing fails on that receiver.
  • Wait 3–5 minutes after the failed push. The receiver records the post-receipt failure in its local audit, the sender's PublisherQueueJob polls the receiver, rewrites the local audit map for that endpoint to a failed status, flips the bundle's delivery strategy to FAILED_ENDPOINTS, and re-queues it.

Deterministic signal that the retry has been triggered (this is the critical part — without this line in the log, the bug is unreachable, regardless of what the UI shows):

INFO  business.PublisherQueueJob - For bundle '<ID>':
INFO  business.PublisherQueueJob - -> Status             : FAILED_TO_SEND_TO_ALL_GROUPS [4]
INFO  business.PublisherQueueJob - -> Re-publish attempts: 1 out of 3
...
INFO  publishing.PublisherAPIImpl - Retrying bundle: <ID>, we don't need to run bundlers again
ERROR pusher.PushPublisher - Cannot invoke "com.dotcms.publisher.business.EndpointDetail.getStatus()" because "epDetail" is null
java.lang.NullPointerException: ...
at com.dotcms.publisher.pusher.PushPublisher.process(PushPublisher.java:205)
Acceptance Criteria
  • PushPublisher.process() no longer throws a NullPointerException when iterating endpoints on a retry where some endpoints have no prior EndpointDetail entry.
  • A null EndpointDetail for an endpoint that was not attempted in a previous pass is treated as "eligible for inclusion in this attempt" (i.e. behaves the same as the first-time-pushing branch at line 197).
  • On retry under DeliveryStrategy.FAILED_ENDPOINTS, the candidate-endpoint list correctly includes (a) endpoints whose previous status was not success, and (b) endpoints that have never been attempted before — without throwing.
  • When the underlying bundle send eventually fails (e.g. all three attempts in MAX_NUM_TRIES exhaust), the per-endpoint "Additional Information" column in the Publishing Queue UI retains the original failure message from the first attempt for that endpoint, not just the most recent retry's outcome.
  • Existing behaviour for environments in "Push to all" mode is unchanged.
  • Existing behaviour for first-time pushes (when endpointsDetail is empty) is unchanged.
dotCMS Version

Confirmed reproducible in:
Latest
26.03.13-02

Severity

High - Major functionality broken

Links

https://helpdesk.dotcms.com/a/tickets/36892

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dotCMS/src/main/java/com/dotcms/publisher/pusher/PushPublisher.java, especially lines 191–218, and trace the retry flow through PublisherQueueJob.java and PublisherAPIImpl.java. Reproduce the retry using the documented two-endpoint Push to one setup and confirm the logged NPE. Done means retries handle missing EndpointDetail entries, preserve failure information, and leave Push to all and first-time behavior unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.