dotCMS / dotCMS/core

Content Drive: field filter re-scans the folder candidate query up to 4x per request

Open
#37,184 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team : Scout Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

When a Content Drive field filter (e.g. a Text field contains-match) is fully resolvable against the search index — no Tag, no Relationship, no workflow, no free-text term — the system still repeats the expensive database candidate-scan query multiple times per request instead of once. On a ~21,000-item folder this took ~4 repeats and made the response ~6x slower than an equivalent search (606ms vs 88ms).

The routing to the index is correct per ADR-0018 (fields flagged searchable on the content type are index-resolved by design — this is not a mis-routing bug). The inefficiency is in the DB-first hybrid chunk loop, which keeps re-scanning in fixed-size chunks (BROWSER_CONTENT_CHUNK_SIZE, default 900) until enough matches survive index narrowing, regardless of match density.

buildPureESQuery (BrowserAPIImpl) already builds nearly everything needed for a single-pass, index-only query for this specific case, but is missing a folder-scoping clause (conFolder, an already-used, low-risk ES field). Adding it and routing to this path only when zero DB-routed criteria, no workflow filter, and no free-text term are present should collapse the repeated scans to one.


Adding some independent measurements that corroborate this issue, gathered testing against a large dataset (~718,174 contentlets):

  • Filtering by a specific searchable field (Content Type: File Asset + File Name filter) took ~2 minutes to return a response, versus ~31.5 seconds for the equivalent global search filter term — roughly 3.8x slower, consistent with the field-filter re-scan behavior described here.
  • Combining a secondary-language filter with the global search filter took ~12 seconds to generate a response.
  • A plain global search in Content Drive took ~25 seconds to return results, versus ~24ms for the same term in the Search All portlet.

These numbers are from a larger dataset than referenced in the issue description, so the relative slowdown compounds further at scale — an additional data point for the "falls within 20% of an equivalent content-search operation" acceptance criterion.

Acceptance Criteria

  • A field filter with no DB-required criteria over a large folder resolves with at most one database candidate-scan, not four
  • Response time for that case falls to within 20% of an equivalent content-search operation
  • Filters combined with Tag, Relationship, workflow, or free-text terms are unaffected — same result sets, same database-first resolution, no behavior change
  • Result sets for Text, Date-range, Multi-select, and Category field filters are unchanged from current behavior
  • The mandatory read-permission filter continues to run, sourced from the database, on every request
  • No dedicated feature flag — ships as default behavior; the general search-strategy config remains the fallback escape hatch

Coordination note

Shares the underlying candidate-scan query with #37183 (spike on that query's plan instability). Whatever #37183 changes about that query must continue to work correctly when invoked at most once per request, as this fix will do for its case.

Parent Epic

#36814

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in BrowserAPIImpl at buildPureESQuery and trace the DB-first hybrid chunk loop, including BROWSER_CONTENT_CHUNK_SIZE and the conFolder field. Compare the eligible no-DB-criteria path with cases containing Tag, Relationship, workflow, or free-text terms, then verify unchanged result sets, mandatory database permission filtering, and at most one candidate scan against the acceptance criteria; coordinate with #37183.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
backend, performance, search
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.