Spike: Single-JVM multi-tenant (multi-schema by DNS) — tenant-propagation surface audit

Research Question

Can dotCMS be made truly multi-tenant — a single installation/JVM serving multiple Postgres schemas selected by the request's DNS hostname — and if so, how do we scope detached work (background threads, Quartz jobs, pub/sub listeners, startup tasks) to the correct tenant when there is no request to key off?

Request-path tenant resolution is straightforward (resolve tenant from Host header → bind schema). The open question this spike answers: what is the propagation surface — every place that today assumes one tenant per JVM and would need an explicit TenantContext set/propagated — and is the effort a project or a moonshot?

Timebox

8h

Acceptance Criteria

• Audit the codebase across the detached-work dimensions (threads/executors, Quartz, cache regions, non-request DB access, pub/sub/cluster, static/singleton state)
• Produce a sized propagation-surface map (sites, high-risk count, chokepoint coverage) — see docs/multi-tenant-propagation-audit.md
• Verify the highest-leverage unknown: is the cache layer a one-place fix or a per-impl slog?
• Identify the silent-corruption hotspots that dictate fail-loud design
• Recommend a sequencing / feasibility slice
• Decision: greenlight the single-JVM model vs. stay process-per-tenant (depends on tenant density — see doc §1)
• If greenlit, file follow-up issues for the spine + each scattered-work cluster

Context

We explored switching dotCMS from process-per-tenant to a single-JVM, multi-schema model. Today the saving grace is JVM = tenant: the process boundary is the tenant context, so background threads, Quartz, pub/sub, and caches inherit the right tenant for free. The moment one JVM serves many schemas, that free isolation disappears and tenant context must be manufactured on every detached code path via a TenantContext ThreadLocal that is set on every entry and propagated to all detached work.

A 6-dimension fan-out audit found 259 sites, 140 high-risk.

Verdict: a project, not a moonshot — but with a hard floor of irreducible scattered work.

The chokepoints that carry the bulk (fix ~6–8, cover >half the high-risk surface)

• DbConnectionFactory.getConnection() / HibernateUtil — bind schema (SET search_path) to TenantContext → covers all 164+ DB call sites
• ChainableCacheAdministratorImpl.put/get/remove/flushGroup — verified one-place fix: prefix the group at the admin funnel (providers partition storage by group; the cross-cluster inval message is built in the same layer and inherits the prefix). Not 43 files.
• DotConcurrentFactory / DotSubmitter — capture/restore decorator → 63 of 84 async sites
• Quartz DotJob/DotStatefulJob + QuartzUtils.scheduleTask() — tenant in/out of JobDataMap
• CacheTransportTopic.notify() (+ tenant on DotPubSubEvent) — covers Postgres/JDBC/Redis listeners
• PublicCompanyFactory.getDefaultCompanyId / LanguageAPI.getDefaultLanguage — the 328 getDefault* calls bottom out here

The irreducible scattered work (the real cost driver — no central fix)

• 21 async bypass sites using raw threads / ForkJoinPool that skip DotSubmitter (ShutdownCoordinator, JobQueueManagerAPIImpl, IntegrityResource, TimeMachineAjaxAction, PopulateContentletAsJSONUtil, BrowserAPIImpl)
• ~35 Quartz jobs each needing a per-tenant-vs-system scope decision (DropOldContentVersionsJob, EscalationThread, UpdateRatingThread, CleanUnDeletedUsersJob, PublisherQueueJob, …)
• ~30 static caches / ThreadLocals (DotTemplateTool, WebAppPool, CustomFieldType, UserPreferencesFactory, app-secret/SAML key derivation off getDefaultCompany())
• 243 runonce + 10 runalways startup tasks needing a per-tenant iteration loop + per-schema db_version/data_version bookkeeping

Silent-corruption hotspots → mandate a fail-loud design

The TenantContext default must be throw on unset, never "default company," or these fail invisibly:

• PermissionCacheImpl, RoleCacheImpl, UserCacheImpl — cross-tenant privilege escalation / impersonation
• AppsCacheImpl + AppsKeyDefaultProvider / SAMLHelper — one global key encrypting all tenants' secrets
• DbConnectionFactory.getConnection() — unbound schema → background thread writes to whatever schema the pooled connection last held (most dangerous write path)
• System-wide delete jobs (DropOldContentVersionsJob, CleanUnDeletedUsersJob) — a missed scope deletes the wrong tenant's data

Recommended feasibility slice

Build TenantContext (fail-loud) + the two corruption-gating chokepoints (DbConnectionFactory, ChainableCacheAdministratorImpl) + wrap DotSubmitter, then drive one request → async → cache-flush path end to end. The fail-loud throw converts the scattered long tail into failing tests found empirically rather than audited by hand.

Decision input

Worth it only at high tenant density (hundreds–thousands of small/idle tenants, where a JVM-per-tenant's baseline RAM dominates). At tens of substantial tenants, process-per-tenant's overhead is noise next to the engineering cost and corruption risk — don't.

Links

• Full audit + sizing table + sequencing: docs/multi-tenant-propagation-audit.md (committed with this issue)

Start with docs/multi-tenant-propagation-audit.md and review its sizing, chokepoint coverage, and irreducible scattered-work sections. Validate the feasibility recommendation against tenant density and the listed corruption hotspots, then record the process-per-tenant versus single-JVM decision. If single-JVM is approved, file follow-up issues for the TenantContext spine and each scattered-work cluster.