dotCMS / dotCMS/core

Denormalize base_type onto identifier table for query performance

Open
#35,162 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

We can improve our db lookup/query performance when querying the db for identifiers of a specific base type like navigation or content drive.

This will be done in 2 PRs. The first does the data model changes, the second changes the SQL we use to query. The data model changes will be in code for at least a month before we merge the code that does the query changes.

Acceptance Criteria
  • startup task that add columns, index and indexes
  • remove unneeded column joins
  • Pass tests and don't break starter loads or push publishing bundes
  • 2 prs
Priority

None

Additional Context

We have data that shows on a loaded site, these type of queries are some of the heaviest - here are the top 3 querys for total_exec_time for one customer

query           | select cvi.working_inode as inode  from contentlet_version_info cvi, identifier id, structure struc, contentlet c  where cvi.identifier = id.id and struc.velocity_var_name = id.asset_subtype and   c.inode = cvi.working_inode and cvi.variant_id=$3  and struc.structuretype in ($4 , $5)  and (id.host_inode = $1)  and id.parent_path=$2  and c.show_on_menu = $6 and cvi.deleted = $7 order by  c.mod_date asc
calls           | 22387276
total_exec_time | 3649630092.113903
mean_exec_time  | 163.02251743866006
-[ RECORD 2 ]---+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
query           | SELECT * FROM QRTZ_EXCL_LOCKS WHERE LOCK_NAME = $1 FOR UPDATE
calls           | 5382189
total_exec_time | 3519143867.950299
mean_exec_time  | 653.8499238786048
-[ RECORD 3 ]---+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
query           | select cvi.live_inode as inode  from contentlet_version_info cvi, identifier id, structure struc, contentlet c  where cvi.identifier = id.id and struc.velocity_var_name = id.asset_subtype and   c.inode = cvi.live_inode and cvi.variant_id='DEFAULT'  and struc.structuretype in (4)  and (id.host_inode = $1)  and id.parent_path=$2  and cvi.deleted = 'false' order by  c.mod_date desc
calls           | 23572980
total_exec_time | 1129625881.7285948
mean_exec_time  | 47.92036822364463

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the identifier table and data model, then compare them with the three example queries in the issue. Define the startup task for the new columns and indexes, verify the data-model and query changes can be split into two PRs, and run the tests plus starter-load and push-publishing checks to confirm the work is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.