dotCMS / dotCMS/core

Site Search: sitesearch_audit.path varchar(500) overflows with long exclude/include path lists, silently breaking incremental indexing

Open
#36,706 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

OKR : Customer Support Team : Maintenance Type : Defect
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Describe the bug

The sitesearch_audit.path column is defined as varchar(500). When a Site Search job is configured with a long list of Include/Exclude paths (combined string longer than 500 characters), the audit insert fails, the exception is caught and only logged (SiteSearchJobImpl: can't save audit data), and the job otherwise reports success.

Because the audit row is the checkpoint for incremental runs (SiteSearchJobImpl only runs incrementally when a recent audit row exists, and uses its fire_date as the delta start date), the next scheduled run finds no audit record and silently falls back to a full index rebuild instead of an incremental update.

To Reproduce
  1. Site Search → Job Scheduler → Paths → Include or Exclude
  2. Enter path patterns whose combined length exceeds 500 characters
  3. Schedule and run the job
  4. Check the dotCMS log:
ERROR job.SiteSearchJobImpl: can't save audit data
com.dotmarketing.exception.DotDataException: ERROR: value too long for type character varying(500)
"SQL": ["insert into sitesearch_audit (job_id,job_name,fire_date,incremental,start_date,end_date,host_list,all_hosts, lang_list,path,path_include,files_count,pages_count,urlmaps_count,index_name) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"]
  1. Run the job again — it performs a full rebuild rather than an incremental update, because no audit row was persisted.
Expected behavior

Long include/exclude path lists should not break audit persistence or incremental indexing. Suggested fix:

  • Change sitesearch_audit.path to text (consider host_list and lang_list too — both are also varchar(500))
  • Additionally, surface the audit-save failure (job error/warning) instead of only logging it, and/or validate the path string length in the Job Scheduler UI
Affected versions

Confirmed by a customer on 26.6.1. The column definition is unchanged in current main (dotCMS/src/main/resources/postgres.sql, create table sitesearch_audit ... path varchar(500) not null).

Known workaround

Manually widen the column (e.g. alter table sitesearch_audit alter column path type text;) — this was the workaround suggested by engineering in #19706. Note that a fresh database created from the stock schema regresses to varchar(500).

Prior reports
  • #8518 (2019 — proposed changing the column to text; closed by the stale bot without a fix)
  • #19706 (2022 — identical symptom and stack trace; closed by the stale bot without a fix)
  • Related feature request: #19707
Related Freshdesk ticket

https://dotcms.freshdesk.com/a/tickets/38328

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dotCMS/src/main/resources/postgres.sql and SiteSearchJobImpl, then reproduce the long Include/Exclude path case and inspect the audit-save failure. Confirm that audit persistence supports long path lists and that a failed save is surfaced or validated as specified; verify that a subsequent scheduled run retains incremental behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.