Content Drive: field filter re-scans the folder candidate query up to 4x per request
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Description
When a Content Drive field filter (e.g. a Text field contains-match) is fully resolvable against the search index — no Tag, no Relationship, no workflow, no free-text term — the system still repeats the expensive database candidate-scan query multiple times per request instead of once. On a ~21,000-item folder this took ~4 repeats and made the response ~6x slower than an equivalent search (606ms vs 88ms).
The routing to the index is correct per ADR-0018 (fields flagged searchable on the content type are index-resolved by design — this is not a mis-routing bug). The inefficiency is in the DB-first hybrid chunk loop, which keeps re-scanning in fixed-size chunks (BROWSER_CONTENT_CHUNK_SIZE, default 900) until enough matches survive index narrowing, regardless of match density.
buildPureESQuery (BrowserAPIImpl) already builds nearly everything needed for a single-pass, index-only query for this specific case, but is missing a folder-scoping clause (conFolder, an already-used, low-risk ES field). Adding it and routing to this path only when zero DB-routed criteria, no workflow filter, and no free-text term are present should collapse the repeated scans to one.
Adding some independent measurements that corroborate this issue, gathered testing against a large dataset (~718,174 contentlets):
- Filtering by a specific searchable field (Content Type: File Asset + File Name filter) took ~2 minutes to return a response, versus ~31.5 seconds for the equivalent global search filter term — roughly 3.8x slower, consistent with the field-filter re-scan behavior described here.
- Combining a secondary-language filter with the global search filter took ~12 seconds to generate a response.
- A plain global search in Content Drive took ~25 seconds to return results, versus ~24ms for the same term in the Search All portlet.
These numbers are from a larger dataset than referenced in the issue description, so the relative slowdown compounds further at scale — an additional data point for the "falls within 20% of an equivalent content-search operation" acceptance criterion.
Acceptance Criteria
- A field filter with no DB-required criteria over a large folder resolves with at most one database candidate-scan, not four
- Response time for that case falls to within 20% of an equivalent content-search operation
- Filters combined with Tag, Relationship, workflow, or free-text terms are unaffected — same result sets, same database-first resolution, no behavior change
- Result sets for Text, Date-range, Multi-select, and Category field filters are unchanged from current behavior
- The mandatory read-permission filter continues to run, sourced from the database, on every request
- No dedicated feature flag — ships as default behavior; the general search-strategy config remains the fallback escape hatch
Coordination note
Shares the underlying candidate-scan query with #37183 (spike on that query's plan instability). Whatever #37183 changes about that query must continue to work correctly when invoked at most once per request, as this fix will do for its case.
Parent Epic
#36814
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in BrowserAPIImpl at buildPureESQuery and trace the DB-first hybrid chunk loop, including BROWSER_CONTENT_CHUNK_SIZE and the conFolder field. Compare the eligible no-DB-criteria path with cases containing Tag, Relationship, workflow, or free-text terms, then verify unchanged result sets, mandatory database permission filtering, and at most one candidate scan against the acceptance criteria; coordinate with #37183.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- backend, performance, search
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100