dotCMS / dotCMS/core

Analytics: Environment Filtering — Infrastructure: provision environment config per customer and execute the ClickHouse reset

Open
#37,410 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dotCMS : Analytics Team : Falcon Type : New Functionality
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Summary

Provision the two new environment-identity env vars per customer StatefulSet, execute the coordinated ClickHouse data wipe + schema recreate against live infrastructure, and roll out the analytics server upgrade in the same maintenance window.

Parent epic: #37349
Spike: #37107
Depends on: the ClickHouse schema sub-task's finalized DDL (this ticket executes that recreate against real infra, it doesn't design the schema); the epic's data-preservation gate (below) must be resolved before any wipe is executed, per tenant.

Background

Infrastructure itself is already cleared for the underlying request — confirmed in the spike: the Authoring instance already talks to the same analytics service and the same ClickHouse as Production, so there's no new network path or provisioning involved for cross-environment reads. What is new infra work is the environment-identity plumbing and the reset execution itself.

New env vars, both surfaced via infrastructure-as-code StatefulSet changes, mirroring the existing DOT_ANALYTICS_TENANT/PROJECT pattern:

  • DOT_ANALYTICS_ENVIRONMENT — this instance's own value, sourced from the dotcms.cloud/environment Kubernetes label via the downward API (fieldRef). Auto-derived, no manual per-customer entry.
  • DOT_ANALYTICS_AVAILABLE_ENVIRONMENTS — comma-separated sibling list (key::Label per entry, label optional), manually curated by Platform per customer at provisioning time. There is no automatic sibling-discovery mechanism (confirmed in the spike: k8s labels only describe themselves, no infra registry exists for this) — this is an explicit, ongoing maintenance list, not a one-time setup.

The reset itself: per the epic, this is a hard gate, not a task —

Confirm no analytics data needs to be preserved, per tenant and environment, including dotcms.com, dev.dotcms.com, and every onboarded Early Adopter tenant. If any of it must survive, the reset approach is invalid and this epic reverts to the migration plan.

This ticket owns executing the wipe + recreate once that gate is confirmed clear — dropping the existing ClickHouse tables and recreating them per the schema sub-task's DDL, coordinated across every customer environment (not just one). Sequence this with the experiments work (#36763, #37227, and the experiment columns from #37016/#37017) — they share the schema being recreated, and the experiment columns must survive the recreate.

Self-hosted / non-Cloud: explicitly out of scope for this epic. Environment identity comes from Cloud labels (dotcms.cloud/environment); self-hosted has no such label and needs a sane default, which is a separate, not-yet-scoped piece of work.

Scope

  • Add DOT_ANALYTICS_ENVIRONMENT (auto-derived via downward API fieldRef) and DOT_ANALYTICS_AVAILABLE_ENVIRONMENTS (manually curated) to infrastructure-as-code StatefulSet manifests, per customer.
  • Curate the initial DOT_ANALYTICS_AVAILABLE_ENVIRONMENTS list for every existing customer (dotcms.com, dev.dotcms.com, every onboarded Early Adopter tenant).
  • Execute the ClickHouse data wipe + schema recreate against every customer's ClickHouse, once the gate below is confirmed and coordinated with the schema sub-task's finalized DDL and the experiments-columns dependency.
  • Roll out the analytics server (CAEM) upgrade in the same maintenance window as the schema recreate.

Out of scope

  • Designing the recreated schema itself (ClickHouse schema sub-task) — this ticket executes it.
  • Self-hosted/non-Cloud environment identity defaults.
  • Automatic sibling-environment discovery — the available-environments list is hand-curated by design.

Dependencies / Gate

  • Hard gate, per the epic: confirm no analytics data needs preserving, per tenant and environment (dotcms.com, dev.dotcms.com, every EA tenant), before executing any wipe. If any tenant needs preservation, this reverts to the discarded migration plan — do not proceed past that tenant's wipe until resolved.
  • Needs the ClickHouse schema sub-task's DDL finalized.
  • Needs the experiments-columns work (#36763, #37227, #37016, #37017) sequenced so those columns survive the recreate.

Acceptance Criteria

  • DOT_ANALYTICS_ENVIRONMENT and DOT_ANALYTICS_AVAILABLE_ENVIRONMENTS present and correct on every customer StatefulSet
  • Data-preservation gate explicitly confirmed clear, per tenant, before that tenant's wipe runs
  • ClickHouse reset executed across all customer environments with the new schema live
  • Analytics server upgrade rolled out in the same window
  • Experiment columns confirmed intact post-recreate, per tenant

Next step

Run /speckit-specify against this issue to produce the formal spec (customer-by-customer rollout plan, exact maintenance-window sequencing) before execution.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the infrastructure-as-code StatefulSet manifests and run /speckit-specify to produce the customer-by-customer rollout plan. Confirm the data-preservation gate, finalized ClickHouse DDL, and experiment-column sequencing before execution. Done means every customer has both environment variables, the coordinated reset and analytics upgrade are complete, and experiment columns are intact afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, kubernetes
Domain
cloud, databases, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.