Bug: PublisherQueueJob never reaches FAILED_TO_PUBLISH when receiver audit poll fails (token/network)

Problem Statement

When a push-publish bundle ends PushPublisher.process() in FAILED_TO_SEND_TO_ALL_GROUPS or FAILED_TO_SEND_TO_SOME_GROUPS because of a permanent receiver-side failure (invalid PP token returning HTTP 401, or an unreachable host), the bundle never transitions to the terminal FAILED_TO_PUBLISH status. The audit row stays in a non-terminal "pending" state indefinitely, and the PublisherQueueJob Quartz tick logs a stack trace every minute.

Mechanism (file: dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java):

1. After the initial failed send, PushPublisher.process() marks the bundle as FAILED_TO_SEND_TO_ALL_GROUPS (or _SOME_GROUPS). Both codes are in the whitelist returned by PublishAuditAPIImpl.getPendingPublishAuditStatus() (PublishAuditAPIImpl.java:531-547), so the bundle is picked up by PublisherQueueJob.updateAuditStatus on the next tick (PublisherQueueJob.java:238-262).
2. updateAuditStatus calls collectEndpointInfoFromRemote (line 249-250). With a permanently-bad token, getRemoteHistoryFromEndpoint (line 650-669) throws DotDataException — either because Jersey's POST fails or because JsonUtil.getObjectFromJson cannot parse the non-JSON 401 body.
3. The outer sendBundle (line 334-348) declares throws DotDataException and does not catch the call to getRemoteHistoryFromEndpoint. The inner sendBundle's try/catch (line 350-368) sits one frame too deep — it only protects per-row updates after the remote history has been fetched successfully.
4. updateAuditStatus has only try/finally per bundle (line 244-260), no catch. The exception propagates out of the for loop, breaking iteration over the remaining pending bundles. execute() catches Throwable at the outermost level (line 222-225) and logs the error.
5. Because updateBundleStatus (line 387-507) is never reached for the failing bundle, neither numTries nor the MAX_NUM_TRIES safety net at line 408 ever trigger. The bundle stays at FAILED_TO_SEND_TO_ALL_GROUPS forever and the poller fails on it every minute.

Additional collateral damage:

• One poisoned bundle starves the entire updateAuditStatus pass: every subsequent bundle in pendingBundleAudits is skipped that tick because the exception breaks the for loop.
• For the specific 401 case, PushPublisher.updatingPublishingDetailStatus also calls deleteElementsFromPublishQueueTable (line 410), so the bundle isn't re-processed by SQLGETBUNDLESTOPROCESS — no further numTries increments from PushPublisher.process() either.

Impact:

• Bundles that fail with a permanent auth or connection error never reach the terminal FAILED_TO_PUBLISH state.
• Their publishing_queue_audit rows never get cleaned up (updateBundleStatus line 418-419 / deleteElementsFromPublishQueueTable are never invoked).
• Every PublisherQueueJob tick prints a stack trace and stops processing other pending bundles after the bad one.

Related to #34356 (push-publish failure events enhancement) — distinct correctness bug uncovered while reviewing that flow.

Steps to Reproduce

1. Configure a sending environment with at least one Push-Publishing endpoint whose Auth Token is invalid (or whose receiver is unreachable / returns 401 for /api/auditPublishing/getAll).
2. Push any bundle (e.g. a single page) to that environment from the Bundle Uploader or the Push Publishing UI.
3. Observe PushPublisher.process() set the audit row to FAILED_TO_SEND_TO_ALL_GROUPS and fire AllPushPublishEndpointsFailureEvent.
4. Wait ≥ 1 minute and tail the dotCMS logs: each PublisherQueueJob.execute tick logs a DotDataException originating from getRemoteHistoryFromEndpoint, and the bundle's status in publishing_queue_audit never advances past FAILED_TO_SEND_TO_ALL_GROUPS even after numTries would otherwise have exceeded MAX_NUM_TRIES (default 3 from PUBLISHER_QUEUE_MAX_TRIES).
5. Confirm via DB: SELECT bundle_id, status, status_pojo FROM publishing_queue_audit WHERE bundle_id = '<id>' — status stays at 4 (FAILED_TO_SEND_TO_ALL_GROUPS) indefinitely.

Acceptance Criteria

• After MAX_NUM_TRIES failed receiver-poll attempts, the bundle transitions to terminal FAILED_TO_PUBLISH (status code 8) and the queue rows are cleaned up — same behaviour updateBundleStatus line 408-419 already implements for other failure paths.
• A single failing bundle does not abort polling for the remaining pending bundles in the same execute() tick.
• On an exception from getRemoteHistoryFromEndpoint, the endpointTrackingMap for the bundle records a synthetic per-endpoint failure (status FAILED_TO_PUBLISH) — analogous to the existing inner sendBundle catch at lines 362-366 — so updateBundleStatus can run, increment numTries, and reach the MAX_NUM_TRIES safety net.
• Unit/integration test reproducing the loop: a bundle marked FAILED_TO_SEND_TO_ALL_GROUPS with numTries = 1 reaches FAILED_TO_PUBLISH after MAX_NUM_TRIES ticks when the receiver mock throws on /api/auditPublishing/getAll.
• No regression in the happy path: bundles that poll successfully still transition through RECEIVED_BUNDLE / PUBLISHING_BUNDLE / SUCCESS as before.

Suggested fix (sketch — open to discussion):

1. Wrap the per-bundle body of updateAuditStatus's for loop with a try/catch so one bundle's failure can't break the others.
2. Hoist the inner sendBundle try/catch one frame up — around the call to getRemoteHistoryFromEndpoint inside the outer sendBundle — and on exception synthesize the same "failed-remote-group-…" EndpointDetail with Status.FAILED_TO_PUBLISH that the inner catch already produces at lines 362-366. That way endpointTrackingMap captures the failure, updateBundleStatus runs, numTries increments, and after MAX_NUM_TRIES the bundle correctly transitions to FAILED_TO_PUBLISH.

dotCMS Version

latest Evergreen 26.05.11-01

Severity

Medium - Some functionality impacted

Links

NA

Start with dotCMS/src/main/java/com/dotcms/publisher/business/PublisherQueueJob.java, especially updateAuditStatus, sendBundle, and updateBundleStatus, then trace getRemoteHistoryFromEndpoint in PublishAuditAPIImpl and the related PushPublisher flow. Add a regression test using a receiver mock that fails during audit polling, and verify retry exhaustion reaches FAILED_TO_PUBLISH, cleans queue rows, continues processing later bundles, and preserves successful polling behavior.