push-publish: bundles stall permanently when collectEndpointInfoFromRemote returns an empty tracking map

Problem Statement

In PublisherQueueJob.updateAuditStatus(), the method collectEndpointInfoFromRemote() can return an empty endpointTrackingMap without throwing an exception. When this happens, getGroupStats() returns all-zeros and updateBundleStatus() evaluates countGroupPublishing (0) == endpointTrackingMap.size() (0) → true, setting the bundle status to PUBLISHING_BUNDLE (status 12).

Status 12 is excluded from getQueueBundleIdsToProcess(), so the bundle stops being re-sent and numTries never advances. The bundle is polled on every job tick indefinitely but is never finalized — it stalls permanently with no way to recover short of manual DB intervention. Operators see no errors in the log; the bundle just silently disappears from the retry queue.

Steps to Reproduce

1. Create a Push Publishing bundle targeting an environment with one or more endpoints.
2. Enqueue the bundle (or allow it to be sent and fail so it's in FAILED_TO_SEND_TO_ALL_GROUPS).
3. Delete the environment (or all endpoints in the environment) while the bundle is still in the queue.
4. Wait for the next PublisherQueueJob tick.
5. Observe: the bundle transitions to PUBLISHING_BUNDLE (status 12), stops appearing in the send loop, numTries freezes, and no log errors are emitted. The bundle is stuck permanently.

Alternatively (transient race, self-resolving): If every endpoint in every environment has isSending() == true at the exact moment the audit pass runs, the same empty-map path is taken. This variant resolves on the next tick once in-flight transfers complete, but it is still a correctness gap.

Acceptance Criteria

• When collectEndpointInfoFromRemote() returns an empty tracking map AND numTries >= MAX_NUM_TRIES, the bundle is finalized as FAILED_TO_PUBLISH and removed from publishing_queue — not left in PUBLISHING_BUNDLE.
• When collectEndpointInfoFromRemote() returns an empty tracking map AND numTries < MAX_NUM_TRIES, the bundle is left as-is for the next tick (not prematurely finalized).
• A meaningful message is written to the PP log (accessible from the dotCMS admin UI) when the empty-map condition is detected.
• A bundle whose environment was deleted after enqueue does not stall indefinitely — it reaches a terminal state within MAX_NUM_TRIES job ticks.
• No regression in the normal push-publish flow (bundles that succeed reach BUNDLE_SENT_SUCCESSFULLY).
• Integration test covering the deleted-environment scenario.

Root Cause

collectEndpointInfoFromRemote() iterates over the environments mapped to a bundle. When no environments are found (e.g., environment was deleted), the outer loop body never runs and the method returns an empty map with no exception. The caller (updateAuditStatus()) passes this empty map to updateBundleStatus(), which has no guard for the empty case.

Suggested fix — in updateAuditStatus(), before calling updateBundleStatus():

if (endpointTrackingMap.isEmpty()) {
    if (bundleAudit.getStatusPojo().getNumTries() >= MAX_NUM_TRIES) {
        finalizeFailedBundle(bundleAudit);
    }
    // else: transient (all endpoints in-flight) — leave for next tick
    return;
}

dotCMS Version

dotCMS evergreen
26.06.21-01

Severity

Medium - Some functionality impacted

Bundles stall silently with no error — difficult to diagnose. Manual DB intervention required to unblock. Does not affect bundles with reachable endpoints.

Related

• Fixed by PR #36342 / Issue #35999: per-bundle catch for unreachable endpoints (UnknownHostException / ProcessingException). The empty-tracking-map scenario is a separate code path that bypasses that fix — the exception catch in #35999 is never reached because collectEndpointInfoFromRemote() returns normally (no throw) when the environment is deleted.

Links

Original ticket: 37586

Start in PublisherQueueJob.updateAuditStatus() and trace collectEndpointInfoFromRemote(), getGroupStats(), and updateBundleStatus() to understand the empty-map path. Add coverage for a deleted environment and verify retry behavior through MAX_NUM_TRIES, terminal failure, PP logging, and the normal successful publishing flow.