dotCMS / dotCMS/core

ES→OS: endpoint to audit and rewrite VTL that calls the deprecated $estool.esSearch / $estool.esRaw

Open
#37,520 3 comments 0 reactions 1 assignee View on GitHub

@fabrizzio-dotCMS is already working on this.

Since Sep 11, 2026.

OpenSearch Team : Scout Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

Customers migrating from Elasticsearch to OpenSearch 3 have to find and rewrite every Velocity
template that calls the two deprecated search methods, $estool.esSearch(...) and
$estool.esRaw(...). Today this is entirely manual, and it is the single hardest part of the
migration for a customer to do alone:

  • The code is spread across four places — templates and containers stored in the database, file
    templates/containers in the assets tree, widget code and custom-field Velocity inside content type
    definitions, and Velocity script steps inside workflow actions. Nothing but the files can be
    grepped.
  • The failure is silent. These templates keep working through Phase 1 (reads still come from
    Elasticsearch) and break in Phase 2, and they break by rendering less, not by erroring.

This task adds one role-gated endpoint that both reports what needs changing and, when explicitly
instructed, applies the mechanical part of the change with a manifest that can be reverted.

Endpoint modes
Mode Behaviour
GET /api/v1/index/migration/vtl-audit Read-only report: object type, id, name, line, rule, confidence. Touches nothing.
POST /api/v1/index/migration/vtl-audit/_apply with dryRun: true Returns the exact diff it would write. Writes nothing.
POST /api/v1/index/migration/vtl-audit/_apply Writes the working version only of each object — never publishes — and returns a runId plus the manifest (object, rule applied, previous version).
POST /api/v1/index/migration/vtl-audit/_revert with a runId Restores the previous version of every object in that manifest.
Rules

Auto-rewritable (high confidence):

From To
$estool.esRaw( $estool.raw(
$r.took (on a variable bound to a raw response) $r.tookInMillis
$r.hits.getAt(n) $r.hits.hits.get(n)
$estool.esSearch( and the $item.map.field uses of its loop variable $estool.search( and $item.field

Reported, never rewritten:

Found Why it needs a human
$r.toString() used as JSON The neutral response does not serialize to engine JSON; the JSON has to be rebuilt field by field. Highest-risk finding: it fails silently.
$results.hits.maxScore No equivalent on the neutral response; the score has to be read per hit.
esSearch whose result variable escapes the file (#parse, #dotParse, a macro) The rewrite of .map. uses cannot be attributed safely.
Velocity steps in workflow actions Reported with the action id; rewriting is allowed only with an explicit target list.
Design constraints
  1. Only rewrite what is provably local. If the result variable of an esSearch call crosses a
    #parse / #dotParse / macro boundary, the finding is reported, not rewritten. A wrong rewrite
    here fails the same silent way the migration itself does.
  2. Never publish. Writing the working version keeps the live site untouched and uses the
    versioning that templates, containers, content types and file assets already have — restoring the
    previous version is the rollback. Publishing stays a human decision after reading the diff.
  3. The manifest is the only backup we own. Workflow actionlets do not version, so for those the
    previous code is stored whole in the manifest.
  4. Idempotent. A second run finds nothing to do; _revert works as long as the runId manifest
    exists.
  5. Gated and off by default. Same gate as the readiness endpoint — CMS Administrator plus the
    migration role — since this reads and rewrites customer code.

Acceptance Criteria

  • GET .../vtl-audit returns findings from all four sources: database templates and containers,
    file templates/containers, content type field values (widget code and custom fields), and
    Velocity workflow actionlets.
  • Every finding carries object type, id, human-readable name, line, rule, and whether it is
    auto-rewritable or manual.
  • _apply with dryRun: true returns a diff and provably writes nothing.
  • _apply writes only working versions; nothing is published, and the live site renders
    identically immediately after a run.
  • _apply returns a runId whose manifest names every object touched and its previous version.
  • _revert with that runId restores every object to its previous state, including workflow
    actionlets, which have no versions of their own.
  • Re-running _apply on an already-converted installation reports and changes nothing.
  • A template whose esSearch result escapes through #parse or a macro is reported and left
    untouched.
  • toString()-as-JSON and maxScore findings are never rewritten.
  • Both endpoints answer 403 without CMS Administrator plus the migration role.
  • Integration tests cover a fixture with one case per rule, plus an already-migrated fixture that
    must produce zero findings.

Additional Context

The rules come from the customer-facing migration guide's template section, which was verified
against the code:

  • ESContentTool.esSearch returns ESSearchResults holding raw Contentlets; ESContentTool.search
    returns ContentSearchResults<ContentMap>. ContentMap has no getMap(), which is why
    $item.map.field stops resolving after the rename — silently.
  • ContentSearchResponse exposes getTookInMillis(), and SearchHits exposes getHits() /
    getTotalHits(); the old getAt(n) and getMaxScore() have no counterpart.
  • On Elasticsearch the neutral search() delegates to the same esSearch() underneath
    (ESSearchAPIImpl.search), so query handling is unchanged by the rewrite — only the result type is.

Both deprecated methods are marked @Deprecated(forRemoval = true) for removal in v26.08.04.

Related: the deprecation inventory in docs/backend/ES_OS_DEPRECATIONS.md and the plugin audit skill
requested in #37466 — this issue is the Velocity half, and the two should share one rule source
rather than each keeping its own list.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.