Support push publishing of large content sets (100k+) without silent truncation at ~10,000 items
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Problem
When a user runs Select All → Add to Bundle from Content Search on a result set larger than ~10,000 items, only ~9,997 items are actually added to the bundle. The remaining items are silently excluded — the UI gives no indication that the selection was truncated, so the user believes they have selected everything.
This makes it impossible to push publish a large single content type (e.g. 18,000 items — and realistically 50k–120k+) as one operation. Customers must instead manually slice the result set (by modification-date range, folder, or category) into multiple sub-10,000 bundles. That workaround is tedious and error-prone: it's easy to create gaps or overlaps between slices, and it re-processes shared dependencies on every slice.
The result is both a workflow blocker for publishing large amounts of content and a silent data-completeness gap in a core publishing operation — the most concerning part is that content is dropped from the selection with no warning.
(Reported in Freshdesk ticket 37863: a customer with ~18,000 SimpleFaqs on dotCMS v24.12.27 LTS saw Select All add only 9,997.)
Goal
Users should be able to push publish a content set of arbitrary size (e.g. 100,000+ items) reliably as a single logical operation, without manually splitting it and without losing items. Just as importantly, the UI should never silently exclude selected content — if any limit is enforced, the user should be clearly told how many items were selected versus how many matched.
This removes a recurring blocker for large content migrations and environment syncs, eliminates a silent data-loss risk in publishing, and avoids forcing customers toward discouraged global configuration changes just to publish large content types.
Target Personas
- Developer teams
- Content teams
- DevOps teams
- System administrators (dotCMS)
Demo Expectations
In the demo, we will show a content type containing more than 10,000 items (e.g. 50,000). Using Select All → Add to Bundle in Content Search, we will confirm the bundle contains the full matching count rather than ~9,997, and that the UI accurately reflects the number of items selected versus the total matched. We will then push publish the entire set to a receiver and confirm all items arrive, with no silent truncation at any step.
Acceptance Criteria
- Select All on a result set larger than 10,000 adds all matching items to the bundle (after working/live de-duplication), not ~9,997.
- A large single content type (e.g. 100,000+ items) can be push published as one logical operation, without the user manually slicing it into smaller bundles.
- The UI never silently truncates a selection; if any limit applies, the user is clearly informed of the count selected versus the count matched.
- Publishing a large content set does not require globally raising index.max_result_window.
User Stories
- As a content editor, I want to select all items of a large content type and add them to a bundle in one action, so that I can push publish the entire set without manually splitting it.
- As a dotCMS administrator, I want push publishing to handle content types with 100,000+ items reliably, so that large migrations and environment syncs complete without silent data loss.
- As a content editor, I want a clear warning when a selection exceeds a limit, so that I never unknowingly publish an incomplete set.
Links
Freshdesk ticket 37863
Related issue #19527 — Rely on Elasticsearch index.max_result_window to dictate max query results
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce Select All → Add to Bundle in Content Search with more than 10,000 matching items, then review related issue #19527 and the selection and push-publishing entry points involved. Done means the full matching set is handled as one logical operation, no items are silently truncated, counts are accurate, and publishing does not require raising index.max_result_window.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- backend, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100