Site Search: sitesearch_audit.path varchar(500) overflows with long exclude/include path lists, silently breaking incremental indexing
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Describe the bug
The sitesearch_audit.path column is defined as varchar(500). When a Site Search job is configured with a long list of Include/Exclude paths (combined string longer than 500 characters), the audit insert fails, the exception is caught and only logged (SiteSearchJobImpl: can't save audit data), and the job otherwise reports success.
Because the audit row is the checkpoint for incremental runs (SiteSearchJobImpl only runs incrementally when a recent audit row exists, and uses its fire_date as the delta start date), the next scheduled run finds no audit record and silently falls back to a full index rebuild instead of an incremental update.
To Reproduce
- Site Search → Job Scheduler → Paths → Include or Exclude
- Enter path patterns whose combined length exceeds 500 characters
- Schedule and run the job
- Check the dotCMS log:
ERROR job.SiteSearchJobImpl: can't save audit data
com.dotmarketing.exception.DotDataException: ERROR: value too long for type character varying(500)
"SQL": ["insert into sitesearch_audit (job_id,job_name,fire_date,incremental,start_date,end_date,host_list,all_hosts, lang_list,path,path_include,files_count,pages_count,urlmaps_count,index_name) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"]
- Run the job again — it performs a full rebuild rather than an incremental update, because no audit row was persisted.
Expected behavior
Long include/exclude path lists should not break audit persistence or incremental indexing. Suggested fix:
- Change
sitesearch_audit.pathtotext(considerhost_listandlang_listtoo — both are alsovarchar(500)) - Additionally, surface the audit-save failure (job error/warning) instead of only logging it, and/or validate the path string length in the Job Scheduler UI
Affected versions
Confirmed by a customer on 26.6.1. The column definition is unchanged in current main (dotCMS/src/main/resources/postgres.sql, create table sitesearch_audit ... path varchar(500) not null).
Known workaround
Manually widen the column (e.g. alter table sitesearch_audit alter column path type text;) — this was the workaround suggested by engineering in #19706. Note that a fresh database created from the stock schema regresses to varchar(500).
Prior reports
- #8518 (2019 — proposed changing the column to
text; closed by the stale bot without a fix) - #19706 (2022 — identical symptom and stack trace; closed by the stale bot without a fix)
- Related feature request: #19707
Related Freshdesk ticket
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with dotCMS/src/main/resources/postgres.sql and SiteSearchJobImpl, then reproduce the long Include/Exclude path case and inspect the audit-save failure. Confirm that audit persistence supports long path lists and that a failed save is surfaced or validated as specified; verify that a subsequent scheduled run retains incremental behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100