dotCMS / dotCMS/core

Refactor Unique Field Data Migration

Open
#31,313 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

  • #31338 by @jdotcms — closed without merging
Team : Falcon Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Parent Issue

No response

Task

Currently unique fields does the most of the migration work based on db; this means the start up is lock until the query gets done for a while.
As a Saas we need to do the migration in background, be not eager on resources (lock tables for a while or use intensive gpu), also the tracking is not possible since the query run as an one unit, the process one by one give us more logs and granularity about any error for a single contentlet (for instance if a contentlet unique field generations fails, we can track and continue with others)
1) The proposal will use a Thread pool with two thread max, in order to run 2 content types at the same time.
2) The proposal will have more logs, so we can see the progress or how is it going
3) The proposal could be fire automatically or by demand, it allow us to do not need to restart the server in case we need to dry run and re-run (aka run by rest only ADMIN)
4) The proposal will use a batches of 100 contentlets per read per thread to avoid the intensive usage of the cpu and memory
5) In order to avoid the starvation, we need to sleep the two threads any time a batch is processed by one second (configurable) this will allow other threads in competition to do their work

Proposed Objective

Core Features

Proposed Priority

Priority 3 - Average

Acceptance Criteria

No response

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the unique-field data migration entry point and reviewing the related pull request, #31338, for existing work. Compare the implementation with the proposal for bounded background concurrency, batch processing, configurable pauses, progress logging, and administrator-triggered runs; the issue provides no acceptance criteria or named tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.