Refactor Unique Field Data Migration
Nobody has claimed this yet.
- #31338 by @jdotcms — closed without merging
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Parent Issue
No response
Task
Currently unique fields does the most of the migration work based on db; this means the start up is lock until the query gets done for a while.
As a Saas we need to do the migration in background, be not eager on resources (lock tables for a while or use intensive gpu), also the tracking is not possible since the query run as an one unit, the process one by one give us more logs and granularity about any error for a single contentlet (for instance if a contentlet unique field generations fails, we can track and continue with others)
1) The proposal will use a Thread pool with two thread max, in order to run 2 content types at the same time.
2) The proposal will have more logs, so we can see the progress or how is it going
3) The proposal could be fire automatically or by demand, it allow us to do not need to restart the server in case we need to dry run and re-run (aka run by rest only ADMIN)
4) The proposal will use a batches of 100 contentlets per read per thread to avoid the intensive usage of the cpu and memory
5) In order to avoid the starvation, we need to sleep the two threads any time a batch is processed by one second (configurable) this will allow other threads in competition to do their work
Proposed Objective
Core Features
Proposed Priority
Priority 3 - Average
Acceptance Criteria
No response
External Links... Slack Conversations, Support Tickets, Figma Designs, etc.
No response
Assumptions & Initiation Needs
No response
Quality Assurance Notes & Workarounds
No response
Sub-Tasks & Estimates
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the unique-field data migration entry point and reviewing the related pull request, #31338, for existing work. Compare the implementation with the proposal for bounded background concurrency, batch processing, configurable pauses, progress logging, and administrator-triggered runs; the issue provides no acceptance criteria or named tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100