dotCMS / dotCMS/core

Content Drive: folder copy, async job endpoint and frontend wiring

Open
#37,062 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dotCMS : Rest API dotCMS: Content Drive Team : Scout Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

Folder copy over a Content Drive multi-select, end to end: the asynchronous backend endpoint and the Content Drive wiring that drives it. One ticket, one operation, one deliverable — copy works in the product when this closes.

Re-split 2026-09-14. This ticket previously carried the backend for copy and bulk delete, with the frontend for copy, delete and move all sitting on #37063. The three tickets are now one per operation, each carrying its own backend and frontend, so each can be handed to one developer and finished without waiting on a sibling. No design decision changed — copy is still an async job for the reasons recorded below. Bulk delete is now #37063, move is #37165.

Why this cannot be a synchronous REST call

Copy is a single unbounded transaction over a recursive walk:

Entry point What runs inside that one transaction
FolderAPIImpl.copy @WrapInTransaction (FolderAPIImpl.java:369,387) FolderFactoryImpl.copy recurses over file assets (:501), pages (:516), links (:539) and child folders (:550)

Copying a folder holding thousands of assets holds one long transaction and times out at the proxy, and a multi-select multiplies it by the size of the selection. An earlier version of this work proposed a synchronous endpoint and designed around the expected timeout ("per-source transactions at least mean a timeout does not lose the folders already done"). Designing around an expected timeout is an async design with none of the machinery, so it is built as one.

Per-folder child lists are also loaded whole rather than paged, so peak memory is bounded by the widest single folder, not by the subtree. That is survivable; the transaction is the part that is not.

The job-queue framework already exists

No new infrastructure. com.dotcms.jobs.business is complete and Postgres-backed: JobProcessor, @Queue, Cancellable, ProgressTracker, retry policies, PostgresJobQueue, AbandonedJobDetector, RealTimeJobMonitor.

JobQueueResource (/v1/jobs) already exposes everything the caller needs after submission:

  • GET /v1/jobs/{jobId}/status
  • POST /v1/jobs/{jobId}/cancel
  • GET /v1/jobs/{jobId}/monitor, an SSE EventOutput
  • paginated active / completed / successful / failed / canceled / abandoned

Closest precedents to copy: CleanAssetsJobProcessor and FixAssetsJobProcessor (@Queue + Cancellable + AtomicBoolean cancel flag + progress reported only when the rounded percentage changes + a summary in getResultMetadata). ImportContentletsProcessor is the older precedent.


Backend

Proposed contract

A domain endpoint that enqueues and returns immediately, following the _import precedent of a domain resource wrapping the job queue rather than making callers post to the generic /v1/jobs/{queueName}:

POST /api/v1/assets/folders/_copy   -> 202, queue "folderCopy"
{
  "sourcePaths": ["//demo.dotcms.com/projects/alpha/", "//demo.dotcms.com/projects/beta/"],
  "destinationPath": "//demo.dotcms.com/archive/"
}

Answers 202 with the job handle:

{ "entity": { "jobId": "e6d9bae8-657b-4e2f-8524-c0222db66355", "statusUrl": "/api/v1/jobs/e6d9bae8-.../status" } }

The body is shared with move (#37165) deliberately, so the two folder relocation operations stay symmetrical.

Validation that can be done at submission time still uses status codes: a malformed body or an unresolvable destination is 400/404 before a job is created. Per-source outcomes are the job's business.

Pieces to build, all following patterns already in WebAssetResource:

Piece Follow
Immutable request form AbstractFolderDeletionRequestForm: @Value.Immutable, @JsonSerialize/@JsonDeserialize, @Schema with RequiredMode.REQUIRED and examples
Path to folder resolution folders/_delete (WebAssetResource.java:314) resolves assetPath; do the same for sources and destination
Permission gate WebResource.InitBuilder as in deleteFolder, plus READ on source and WRITE / CAN_ADD_CHILDREN on destination
OpenAPI @Operation annotations, then regenerate openapi.yaml
Per-path results move into the job result, they do not disappear

JobProcessor.getResultMetadata(Job) returns Map<String, Object>, so the per-path contract survives verbatim, just relocated out of a synchronous response body:

{
  "successCount": 3,
  "failCount": 2,
  "results": [
    { "path": "//demo.dotcms.com/projects/alpha/", "success": true },
    { "path": "//demo.dotcms.com/projects/beta/", "success": false, "error": "NAME_COLLISION", "message": "..." }
  ]
}

Each source folder is its own transaction, so one collision does not roll back the folders already copied. This shape is shared with #37063 and #37165 — the three must not diverge.

The real work is chunking, not the endpoint

Wrapping this in a job does not by itself fix the transaction. A job holding one 40-minute transaction is still one 40-minute transaction; it just stops timing out at the proxy. Meaningful progress, working cancellation and bounded memory all require chunking inside the recursion, which means changing FolderFactoryImpl, not only adding a JobProcessor. Estimate this ticket on that, not on the endpoint.

Cancellation semantics

A cancelled copy leaves the partial subtree in place at the destination, matching the CleanAssetsJobProcessor precedent — the user can delete it. Cancellation may land mid-folder. Two jobs copying overlapping subtrees must not run concurrently; the second submission is rejected with a readable reason rather than queued.


Frontend

Content Drive's action center operates on contentlets. Folders are selectable in the table but cannot be copied. This is the wiring.

#33468 already describes the intended destination-picker UX (a picker, not drag-and-drop, with a toast on completion), so this is the implementation of that for folder copy.

The job-progress primitive comes from #37166

Nothing in core-web calls /v1/jobs today, so a reusable primitive has to be built. #37166 (bulk file upload) owns it, because #32356 asks for the richer version (per-file progress and per-file cancel) and a primitive that satisfies upload satisfies the folder actions too. This ticket consumes it. Whichever of #37062 / #37063 / #37165 / #37166 lands first builds it to #37166's stated requirements and the rest reuse it unchanged — do not build it four times.

The flow: submit and get a jobId, follow the job, render progress, handle the terminal state, then report the result and refresh.

Following the job has two options and the cheap one is available:

  • GET /v1/jobs/{jobId}/monitor is an SSE EventOutput, and because it is a GET, native EventSource works with no shim. Worth stating explicitly: dot-content-drive-action-center.component.ts:126 records that the legacy _bulkfire SSE path was skipped precisely because "native EventSource cannot POST a body". That objection does not apply here — submission is a normal POST returning a jobId, monitoring is a separate GET.
  • GET /v1/jobs/{jobId}/status polling is the fallback if SSE proves awkward behind the proxy.

Things the primitive has to get right, which are easy to miss:

  • A job outlives the component. Navigating away, or closing the dialog, does not cancel it. Either the outcome stays discoverable when the user comes back, or the UI must say plainly that it will not be. Do not silently lose a running job's result.
  • Cancellation is exposed over POST /v1/jobs/{jobId}/cancel, with copy wording: cancelling leaves a partial copy at the destination.
Bulk means per-path results, not a boolean

The toast must report the server's counts, not the selection size — the same rule the existing quick actions follow. A per-folder failure has to name the folder and say why: name collision, permission, unresolvable path.

Copying a folder copies everything in it. The grid and the sidebar tree both refresh once the job completes, since they reload separately.

Acceptance Criteria

Submission

  • POST /api/v1/assets/folders/_copy accepts several sourcePaths plus one destinationPath, enqueues a folderCopy job and returns 202 with jobId and statusUrl
  • An immutable request form follows the AbstractFolderDeletionRequestForm pattern, with @Schema descriptions and examples on every property
  • A malformed body returns 400; an unresolvable destination returns 404, before any job is created
  • A caller with no rights to enqueue, or no WRITE / CAN_ADD_CHILDREN on the destination, gets 403 at submission
  • A destination resolving to a folder or to a site root both work; both FolderAPI.copy overloads already exist

Job execution

  • The processor is @Queue-annotated, implements Cancellable, and reports progress through Job#progressTracker() only when the rounded percentage changes
  • Each source folder runs in its own transaction; a failure on one does not roll back sources already completed nor abort the remaining ones
  • The recursion is chunked so that a folder with tens of thousands of descendants does not run in a single transaction and does not hold the whole subtree in memory
  • getResultMetadata returns successCount, failCount and a per-path results array; every failure carries a machine-readable error and a human-readable message
  • Copying a folder onto itself is a per-path failure naming the folder
  • Copying a folder into one of its own descendants is a per-path failure naming both folders
  • A name collision at the destination is a per-path failure, never a silent success
  • A per-source permission failure (no READ on source, no CAN_ADD_CHILDREN on destination) is reported per path
  • An unresolvable source path is a per-path failure, not a request-level error
  • The self-target and descendant-target guards and the index refresh live in FolderAPI.copy, not in the processor or the resource; BrowserAjax.copyFolder (BrowserAjax.java:963-970) is updated to rely on them so the logic exists once
  • Cancelling a copy leaves the partial subtree in place and the job result records how far it got
  • A submission overlapping an in-flight job's subtree is rejected with a readable reason rather than queued

Frontend

  • Copy appears for folders in Content Drive and opens a destination picker consistent with #33468
  • Copy follows its job through the shared primitive specified on #37166; no second implementation is added
  • If this ticket lands before #37166, it builds the primitive to #37166's stated requirements and the other tickets reuse it unchanged
  • A running copy job is not cancelled by component teardown or navigation, and its outcome remains discoverable afterwards
  • Progress renders while the job runs
  • The completion toast reports the server's successCount / failCount, never the selection size
  • A folder that could not be copied is named in the result along with why, not silently dropped
  • A name collision at the destination surfaces as a readable message naming the conflicting folder
  • Cancellation wording states that a partial copy is left at the destination
  • The grid and the sidebar tree both refresh once the job completes
  • Copy is hidden or disabled for folders the user has no permission to copy, and for destinations they cannot add children to
  • Only one action runs at a time, matching the existing actionExecution guard in the store

Docs and tests

  • The @Operation description states that the operation is asynchronous, names the folderCopy queue, and points at the /v1/jobs/{jobId} status, cancel and monitor endpoints
  • openapi.yaml regenerated from the annotations and committed alongside the Java changes
  • Integration tests cover: happy path over several folders, mixed partial failure, self-target, descendant-target, name collision, permission denied, folder-vs-site-root destination, cancellation mid-run, and a subtree large enough to exercise more than one chunk
  • Jest specs cover: progress updates, success, partial failure, job failure, cancellation, permission-denied, and teardown while a job is in flight
Priority

Medium

Additional Context

History of the decisions on this ticket, so they are not relitigated.

  • 2026-08-24 — copy became async. It previously proposed three synchronous endpoints (_copy, _move, _bulkdelete) and recorded async as "deliberately out of scope". That call did not survive review; the reasoning is under Why this cannot be a synchronous REST call.
  • 2026-08-24 — move was split out to #37165. Move does not belong with copy because its cost profile is different: FolderFactoryImpl.move (:581) mints a new folder identifier and re-points every contentlet individually, which copy does not do.
  • 2026-09-14 — re-split by operation. Backend and frontend for copy now live together here.

The shipped single _delete (WebAssetResource.java:314) is untouched by this ticket. It takes one assetPath (AbstractFolderDeletionRequestForm:24) and is what the Content Drive context menu already uses. Bulk delete is #37063.

Two adjacent folder operations need no backend work at all:

  • Add to bundle: folder identifiers already flow through the existing path. PublisherAPIImpl.java:251 resolves an unrecognised id via FolderAPI.find() and tags it PusheableAsset.FOLDER, and Folder.setIdentifier() (Folder.java:255) keeps inode == identifier. Both the legacy addToBundle servlet and POST /api/v1/bundles/assets accept folder ids today. Covered on #35161.
  • Push publish: same resolution path, and PublisherAPIImpl enforces PERMISSION_PUBLISH, reporting a denial as a per-asset entry in errorMessages. Covered on #35161.

Publish all is not being carried forward. Previously listed as a bulk fire with +conFolder:<folderInode>; dropped as unsupported. PublishFactory.publishAsset(Folder, ...) (PublishFactory.java:206) publishes contentlets directly and bypasses workflow schemes entirely, so the alternative is worse than the gap. Anything under a folder can still be published by selecting it and firing a workflow action.

Related: #37063 (bulk folder delete, end to end — same job-result contract), #37165 (folder move, end to end — shares the request body shape), #37166 (owns the job-progress primitive this ticket consumes, and reuses this ticket's job-result contract), #33468 (move/copy destination picker UX), #35161 (folder push publish family), #32302 (folder CRUD backend, original source of this carve-out), #32357 (the async folder move/copy feature this sits under), #36046 (bulk delete/download endpoints for publishing), #33999 (parent epic).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with FolderAPIImpl.copy and WebAssetResource.java around folders/_delete, then compare CleanAssetsJobProcessor and FixAssetsJobProcessor for queue behavior. For the UI, inspect dot-content-drive-action-center.component.ts and the existing actionExecution store guard, plus the shared primitive described in #37166. Done means the asynchronous folder-copy flow, chunking, per-path results, cancellation, refreshes, OpenAPI, and acceptance tests all work together.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, typescript
Domain
api, backend-api-design, frontend, full-stack
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.