EPIC: Telemetry API Refactoring - Production MVP & F... | dotCMS/core

Telemetry API Refactoring - Production MVP & Framework

Epic Goal

Deliver production-ready Usage Portlet with minimal metrics set (10-15 metrics) loading in < 5 seconds, with performance visibility and framework for progressive enhancement.

Business Value

Immediate: Functional Usage Dashboard for customer insights
Performance: Visible metric timing identifies bottlenecks
Scalability: Framework supports expanding from 15 → 50 → 128 metrics
Quality: Foundation for testing, monitoring, and optimization
Future-Ready: Architecture supports both code-based and configuration-based metrics

Target: 2-Week Sprint (MVP)

Sprint Goal

Working Usage Portlet demonstrating:

✅ 10-15 core metrics (sites, content types, active users, workflows)
✅ Load time < 5 seconds
✅ Performance timing visible per metric
✅ Error handling and graceful degradation
✅ Framework ready for expansion

Architecture Decisions

Profile System: Hybrid Approach

Annotation: @ProfileCapable - Metrics declare profile capability
Configuration: telemetry.default.profile=MINIMAL - Selects active profile
Rationale: Type-safe (enum) + runtime-configurable (properties)
Future-Ready: Works by metric name, supports config-based metrics

Caching: Configuration-Driven

No Annotations: All caching controlled via configuration
Rationale: Works for future config-based metrics, runtime-configurable
Implementation: MetricCacheManager reads from properties
Future-Ready: Works by metric name, supports config-based metrics

Timing: Collector Level

Location: Instrument at MetricStatsCollector, not in metrics
Rationale: Consistent for all metrics, no metric changes needed
Future-Ready: Works for both code-based and config-based metrics

Timeout: Configuration-Driven

Optional Annotation: @Timeout provides hint (can be overridden)
Configuration: Primary source of truth
Rationale: Runtime-configurable, per-metric control
Future-Ready: Works by metric name

Interface Stability

No Changes to MetricType: Interface stays simple
Rationale: Supports both code-based and config-based implementations
Future-Ready: Easy to implement from configuration

Key Design Principles

Name-Based Operations: All infrastructure works by metric name, not class
Configuration-Driven: All behavior controlled via configuration
Separation of Concerns: Clear boundaries between components
Interface Stability: MetricType interface stays simple
Future-Ready: Supports both code-based and config-based metrics

Tracked Sub-Issues (Dependency-Ordered)

✅ Phase 1: CDI Foundation (COMPLETE)

Days 1-2: CDI Foundation

#33979 - CDI Refactoring (4 points) - COMPLETE - ENABLES EVERYTHING ELSE

🔄 Phase 2: Profile System (NEXT - START HERE)

Day 3: Profile System ← FOUNDATIONAL - DO BEFORE CACHING

#33987 - MINIMAL Profile Implementation (2 points) - START HERE
Rationale: Profiles are foundational - caching and other features depend on profile filtering

🔄 Phase 3: Caching System

Day 4: Caching

#33980 - Flexible Caching and Metric Selection (2 points)
Rationale: Caching works better when profiles are in place

🔄 Phase 4: Performance Timing

Day 4-5: Timing Infrastructure

Performance Timing Infrastructure (part of #33980 or new sub-issue)
Rationale: Timing infrastructure needed for performance visibility

🔄 Phase 5: Timeout Handling

Day 5: Timeout

Timeout Handling (part of #33980 or new sub-issue)
Rationale: Timeout handling completes the performance infrastructure

🔄 Phase 6: Cleanup

Day 5: Cleanup

#33985 - Remove API Metrics System (1 point)
Rationale: Cleanup after main features working

Week 2: Enhancement (STRETCH)

Days 6-10: Integration & Testing

#33982 - Micrometer Integration & Testing (4 points)

Future Work (Post-Sprint)

#33981 - Full Performance Diagnostics
#33983 - Code Consolidation with Generics

Why This Order?

CDI FIRST (#33979) enables:

✅ Profile filtering via @ProfileCapable annotation (makes #33987 easy)
✅ Cache injection into metrics (makes #33980 trivial)
✅ Automatic metric discovery (makes #33982 automatic)
✅ Clean dependency injection (better code quality)

PROFILE SYSTEM FIRST (#33987) enables:

✅ Caching can filter by profile
✅ Timing can be profile-aware
✅ Foundation for all other features

Effort Reduction:

Original plan: 15-16 points
CDI-first plan: 13 points
Savings: 3 points + cleaner code!

Sprint Total: 13 points (realistic for 2 weeks)

Week 1 Must Complete: 9 points (Days 1-5)
Week 2 Stretch: 4 points (Days 6-10)

MVP Definition (Must Ship)

Functional Requirements

Core Metrics Display (10-15 metrics):
- Total Sites / Active Sites
- Total Content Types
- Total Live/Working Content
- Total Active Users
- Total Workflows / Active Workflows
- Total Languages
- Total Experiments, URL Maps
Performance Requirements:
- Total load time < 5 seconds
- Per-metric execution time visible
- Timeout handling (2s per metric)
- Cache for slow metrics
Quality Requirements:
- Real data (no mocks/placeholders)
- Error handling (failed metrics don't break UI)
- User-friendly error messages
- Performance warnings (> 2s highlighted)

Technical Foundation

CDI dependency injection (Day 1-2) ✅ COMPLETE
Profile system (MINIMAL, STANDARD, FULL) - Hybrid approach
Timing infrastructure for all metrics - Collector level
Configuration-driven caching - No annotations
Configuration-driven timeout - Optional annotation hint

Core Metrics (MINIMAL Profile)

Critical Metrics (8 required)

Total Sites
Total Active Sites
Total Content Types
Total Live Contents
Total Working Contents
Total Active Users (30 days)
Total Workflows
Total Languages

Valuable Metrics (5 optional)

Total Experiments
Total URL Maps
Total Active Workflows
Total Containers
Total Templates

Selection Criteria

✅ Fast queries (< 500ms)
✅ Single table or simple joins
✅ Clear business value
❌ No complex aggregations
❌ No multi-table subqueries

Success Metrics

Performance (Must Meet)

Usage Portlet loads in < 5 seconds
Each metric executes in < 2 seconds
Timing data visible for diagnostics
Cache hits return in < 100ms (if implemented)

Functionality (Must Meet)

10-15 metrics display real data (no mocks)
Error handling graceful and informative
Slow metrics highlighted in UI
Profile system enables future expansion

Quality (Should Meet)

Integration tests validate performance
Documentation explains configuration
Framework ready for more metrics
Zero regressions in existing functionality
Future-ready for config-based metrics

Architecture Notes

Database-Sourced Gauge Metrics

All metrics query shared database
All replica instances return identical values
Perfect for direct REST API use in Usage Dashboard
Prometheus can scrape any instance (consistent values)
No complex per-instance aggregation needed

Clear Separation of Concerns

Telemetry API (this epic):

Purpose: Product usage analytics
Metrics: All Gauges (point-in-time snapshots)
Collection: Scheduled daily + on-demand REST
Data source: Database queries
Use case: Usage Dashboard, product insights

Micrometer Operational Monitoring (existing):

Purpose: Real-time operational monitoring
Metrics: Gauges, Counters, Timers
Collection: Continuous, real-time
Data source: Application runtime
Use case: DatabaseMetrics, HttpMetrics, CacheMetrics

Future-Proofing

The architecture built in this epic already supports future configuration-based metrics:

✅ Profile system works by name
✅ Caching works by name
✅ Timing/timeout work by name
✅ Configuration-driven behavior
✅ Simple interface (easy to implement from config)

Configuration Reference

Profile Configuration

telemetry.default.profile=MINIMAL

Cache Configuration

telemetry.cache.enabled=true
telemetry.cache.default.ttl.seconds=300
telemetry.cache.max.size=1000
telemetry.cache.metric.COUNT_OF_SITES.enabled=true
telemetry.cache.metric.COUNT_OF_SITES.ttl.seconds=600

Timeout Configuration

telemetry.metric.timeout.seconds=2
telemetry.collection.timeout.seconds=10
telemetry.metric.COUNT_OF_SITES.timeout.seconds=5

Migration Strategy (Long-Term)

Current State

dotCMS instances push telemetry data daily via Quartz job → Configurator microservice → RDS → Grafana.

Future State (Post-Sprint)

dotCMS instances expose telemetry via Prometheus endpoint → Prometheus scrapes → Grafana queries PromQL.

Migration Phases

Phase 2 (Sprint 1-2 - MVP):

✅ Add REST API for Usage Dashboard (direct database queries)
✅ Keep push-based system for backward compatibility
Deliverable: Working Usage Dashboard with current values

Phase 3 (Sprint 3-4 - Post-MVP):

✅ Add Micrometer integration (TelemetryMetricsBinder)
✅ Prometheus scrapes /actuator/prometheus endpoint
✅ Run both push and pull systems in parallel (validation)
Deliverable: Real-time metrics in Prometheus/Grafana

Phase 4 (Sprint 5-6 - Enhancement):

✅ Add Grafana API proxy in dotCMS backend
✅ Usage Dashboard queries Grafana for historical trends
✅ Custom visualizations with Chart.js/D3
Deliverable: Rich Usage Dashboard with 30-day trends

Phase 5 (6+ months - Deprecation):

✅ Validate Prometheus data matches RDS data
✅ Migrate all Grafana dashboards to PromQL
✅ Deprecate and remove push-based system
✅ Decommission Configurator microservice and RDS
Deliverable: Simplified architecture, cost savings ($2K-6K/year)

Risk Mitigation

Technical Risks

CDI migration issues → ✅ Resolved (already complete)
Slow metrics block UI → Timeout handling (2s per metric), skip slow ones
Cache complexity → Configuration-driven, simple implementation
Data accuracy issues → Validation step in testing

Process Risks

Scope creep → Strict MVP definition, clear phases
Discovery delays → 20% time buffer in estimates
Integration issues → Decision points every 2-3 days

Mitigation Strategy

Clear decision points (Day 2, Day 5, Day 7, Day 10)
Fallback options at each phase
Daily progress tracking
Minimum viable fallback plan

Decision Points

End of Day 2 (CDI Foundation)

Question: Is CDI working for MINIMAL metrics?

✅ YES → ✅ COMPLETE - Proceed to Phase 2 (Profile System)
❌ NO → Fallback to static initialization, adjust plan

End of Day 3 (Profile System)

Question: Is profile system working?

✅ YES → Proceed to Phase 3 (Caching)
❌ NO → Extend profile work, adjust timeline

End of Day 5 (Week 1 Complete)

Question: Is Usage Portlet working < 5s?

✅ YES → Proceed to Week 2 (Micrometer)
❌ NO → Extend testing, skip stretch goals

End of Day 10 (Sprint Complete)

Question: Is system production-ready?

✅ YES → Ship MVP, plan next sprint
❌ NO → Identify gaps, extend timeline

Deliverables (End of Sprint)

Code

✅ Working Usage Portlet with 10-15 metrics
✅ CDI-based metric discovery ✅ COMPLETE
✅ Performance timing infrastructure
✅ Profile system (MINIMAL, STANDARD, FULL) - Hybrid approach
✅ Configuration-driven caching
✅ Error handling and timeout logic
✅ Micrometer integration (if time permits)

Documentation

✅ README.md explaining scope and architecture
✅ Configuration guide (profiles, caching)
✅ Adding metrics guide (CDI pattern)
✅ Performance tuning guide
✅ Architecture documentation (future-proofing)

Tests

✅ Integration test (MINIMAL profile < 5s)
✅ Performance validation test
✅ Error handling test

Demo

✅ Working Usage Portlet demonstration
✅ Performance metrics visible
✅ Framework expansion capability

Effort Summary

Sprint Capacity (2 Weeks)

Minimum Viable: 9 points (5 days) - MUST SHIP
Production Ready: 13 points (10 days) - TARGET

Issues Breakdown

Issue	Phase	Effort	Priority	Days	Status
#33979	CDI Foundation	4 pts	Must	1-2	✅ COMPLETE
#33987	MINIMAL Profile	2 pts	Must	3	🔄 NEXT
#33980	Caching	2 pts	Must	4	🔄 After profiles
Timing	Performance Timing	-	Must	4-5	🔄 After caching
Timeout	Timeout Handling	-	Must	5	🔄 After timing
#33985	Remove API	1 pt	Must	5	🔄 After core
#33982	Micrometer	4 pts	Stretch	6-10	🔄 Stretch

Key Principles

CDI First ✅ - Foundation enables everything else (COMPLETE)
Profile System First - Foundation for caching and other features
Ship working MVP in 2 weeks - No compromises
Performance visibility is critical - Show timing, identify problems
Framework over features - Build for expansion
Progressive enhancement - Clear phases, clear decision points
Database-sourced gauges - Simple, consistent, scalable
Configuration-driven - Runtime-configurable, future-ready
Name-based operations - Works for both code-based and config-based metrics
Quality from start - Tests, docs, error handling

EPIC: Telemetry API Refactoring - Production MVP & Framework

Description