dotCMS / dotCMS/core

perf(db): UNLOGGED permission_reference and LZ4 compression on variable-length columns

Open
#35,186 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Summary

Two PostgreSQL storage optimizations that reduce I/O pressure and improve read latency across dotCMS.


1. Convert `permission_reference` to UNLOGGED

`permission_reference` is a pure denormalized permission cache — rows are rebuilt automatically whenever permissions are invalidated. WAL durability is unnecessary for this table.

Change: `ALTER TABLE permission_reference SET UNLOGGED`

Expected benefits:

  • ~2–3× faster INSERT/DELETE throughput during permission rebuilds
  • Reduced WAL volume, lowering I/O pressure and replication lag
  • Smaller checkpoints during mass permission recalculation

Implementation: `Task260403SetPermissionReferenceUnlogged` — checks `pg_class.relpersistence` to skip if already unlogged; idempotent.


2. LZ4 compression on all text / bytea / jsonb columns

PostgreSQL defaults to pglz for TOAST compression. LZ4 decompresses ~3–5× faster than pglz with comparable compression ratios on typical dotCMS data (HTML, JSON, workflow payloads). Read-heavy workloads (content delivery, page rendering, workflow evaluation) pay the decompression cost on every fetch of a TOASTed column.

Scope: 102 columns across all tables including:

  • `contentlet.contentlet_as_json`, `text_area1`–`text_area25`
  • `template.body`, `header`, `footer`, `drawed_body`, `head_code`
  • `workflow_action.metadata`, `structure.metadata`
  • All `bytea` columns (Quartz job data, storage_data)
  • `experiment`, `job`, `publishing_queue_audit`, etc.

Notes:

  • `SET COMPRESSION lz4` only affects future writes; existing TOAST values rewritten lazily on next UPDATE — migration is instant and lock-free
  • Requires PostgreSQL 14+; gracefully skips on older versions
  • `postgres.sql` updated with `COMPRESSION lz4` inline in all `CREATE TABLE` column definitions (fresh installs)

Implementation: `Task260403SetLz4CompressionOnTextColumns` — dynamically queries `pg_attribute` to find all eligible columns not yet using LZ4, covers plugin-added columns automatically.


Branch

`feature/postgres-storage-optimizations`

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Task260403SetPermissionReferenceUnlogged and Task260403SetLz4CompressionOnTextColumns, then inspect postgres.sql for the corresponding CREATE TABLE definitions. Verify how the migrations detect PostgreSQL capabilities, skip already-compliant columns, and handle plugin-added columns. Done means permission_reference is UNLOGGED and eligible text, bytea, and jsonb columns use LZ4 while older PostgreSQL versions are skipped safely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
backend, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.