Analytics: Environment Filtering — Infrastructure: provision environment config per customer and execute the ClickHouse reset
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Summary
Provision the two new environment-identity env vars per customer StatefulSet, execute the coordinated ClickHouse data wipe + schema recreate against live infrastructure, and roll out the analytics server upgrade in the same maintenance window.
Parent epic: #37349
Spike: #37107
Depends on: the ClickHouse schema sub-task's finalized DDL (this ticket executes that recreate against real infra, it doesn't design the schema); the epic's data-preservation gate (below) must be resolved before any wipe is executed, per tenant.
Background
Infrastructure itself is already cleared for the underlying request — confirmed in the spike: the Authoring instance already talks to the same analytics service and the same ClickHouse as Production, so there's no new network path or provisioning involved for cross-environment reads. What is new infra work is the environment-identity plumbing and the reset execution itself.
New env vars, both surfaced via infrastructure-as-code StatefulSet changes, mirroring the existing DOT_ANALYTICS_TENANT/PROJECT pattern:
DOT_ANALYTICS_ENVIRONMENT— this instance's own value, sourced from thedotcms.cloud/environmentKubernetes label via the downward API (fieldRef). Auto-derived, no manual per-customer entry.DOT_ANALYTICS_AVAILABLE_ENVIRONMENTS— comma-separated sibling list (key::Labelper entry, label optional), manually curated by Platform per customer at provisioning time. There is no automatic sibling-discovery mechanism (confirmed in the spike: k8s labels only describe themselves, no infra registry exists for this) — this is an explicit, ongoing maintenance list, not a one-time setup.
The reset itself: per the epic, this is a hard gate, not a task —
Confirm no analytics data needs to be preserved, per tenant and environment, including dotcms.com, dev.dotcms.com, and every onboarded Early Adopter tenant. If any of it must survive, the reset approach is invalid and this epic reverts to the migration plan.
This ticket owns executing the wipe + recreate once that gate is confirmed clear — dropping the existing ClickHouse tables and recreating them per the schema sub-task's DDL, coordinated across every customer environment (not just one). Sequence this with the experiments work (#36763, #37227, and the experiment columns from #37016/#37017) — they share the schema being recreated, and the experiment columns must survive the recreate.
Self-hosted / non-Cloud: explicitly out of scope for this epic. Environment identity comes from Cloud labels (dotcms.cloud/environment); self-hosted has no such label and needs a sane default, which is a separate, not-yet-scoped piece of work.
Scope
- Add
DOT_ANALYTICS_ENVIRONMENT(auto-derived via downward APIfieldRef) andDOT_ANALYTICS_AVAILABLE_ENVIRONMENTS(manually curated) toinfrastructure-as-codeStatefulSet manifests, per customer. - Curate the initial
DOT_ANALYTICS_AVAILABLE_ENVIRONMENTSlist for every existing customer (dotcms.com, dev.dotcms.com, every onboarded Early Adopter tenant). - Execute the ClickHouse data wipe + schema recreate against every customer's ClickHouse, once the gate below is confirmed and coordinated with the schema sub-task's finalized DDL and the experiments-columns dependency.
- Roll out the analytics server (CAEM) upgrade in the same maintenance window as the schema recreate.
Out of scope
- Designing the recreated schema itself (ClickHouse schema sub-task) — this ticket executes it.
- Self-hosted/non-Cloud environment identity defaults.
- Automatic sibling-environment discovery — the available-environments list is hand-curated by design.
Dependencies / Gate
- Hard gate, per the epic: confirm no analytics data needs preserving, per tenant and environment (dotcms.com, dev.dotcms.com, every EA tenant), before executing any wipe. If any tenant needs preservation, this reverts to the discarded migration plan — do not proceed past that tenant's wipe until resolved.
- Needs the ClickHouse schema sub-task's DDL finalized.
- Needs the experiments-columns work (#36763, #37227, #37016, #37017) sequenced so those columns survive the recreate.
Acceptance Criteria
-
DOT_ANALYTICS_ENVIRONMENTandDOT_ANALYTICS_AVAILABLE_ENVIRONMENTSpresent and correct on every customer StatefulSet - Data-preservation gate explicitly confirmed clear, per tenant, before that tenant's wipe runs
- ClickHouse reset executed across all customer environments with the new schema live
- Analytics server upgrade rolled out in the same window
- Experiment columns confirmed intact post-recreate, per tenant
Next step
Run /speckit-specify against this issue to produce the formal spec (customer-by-customer rollout plan, exact maintenance-window sequencing) before execution.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the infrastructure-as-code StatefulSet manifests and run /speckit-specify to produce the customer-by-customer rollout plan. Confirm the data-preservation gate, finalized ClickHouse DDL, and experiment-column sequencing before execution. Done means every customer has both environment variables, the coordinated reset and analytics upgrade are complete, and experiment columns are intact afterward.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, kubernetes
- Domain
- cloud, databases, devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100