[EPIC] CI/CD Pipeline Stability and Developer Visibility

Description

This epic addresses systemic issues affecting developer productivity and CI/CD reliability across the dotCMS engineering organization.

Problems Being Solved

1. Flaky Tests Impact All Developers: Backend integration test flakiness (e.g., ImportUtilTest, BundlePublisherTest) causes failures for unrelated PRs, wasting developer time investigating issues not caused by their changes.

2. Frontend Developer Friction: Frontend developers making quick, small PRs wait 45+ minutes in merge queue for full integration/postman tests that cannot possibly be affected by their changes.

3. Limited Visibility: Developers have poor visibility into:

   • Why their PR is stuck or failing in the merge queue
   • Whether a failure is caused by their code vs. known flaky tests vs. infrastructure
   • Overall CI/CD health trends and patterns

4. Wasted CI Resources: Orphaned merge queue runs continue consuming resources when PRs are reordered or removed, and full test suites run unnecessarily for frontend-only changes.

5. No Incident Management Process: When catastrophic failures block the merge queue, there's no clear process for developer communication, emergency fixes, or state management.

Proposed Solutions

• Path-based test filtering in merge queue (ADR-0013) - reduce frontend merge time from 45min to 10-15min
• Orphaned run cancellation - save hours/day of CI resources
• AI-powered failure analysis - categorize failures and link to known issues
• CI/CD health metrics - weekly reports tracking failure rates, queue times, and trends
• Circuit breaker system - manage catastrophic failures with clear state control
• PR categorization - separate metrics for frontend vs. backend PRs

Customer Value

• Faster frontend velocity: 75% reduction in merge queue time for frontend PRs
• Reduced developer frustration: Clear communication when failures are not their fault
• Data-driven decisions: Metrics to prioritize flaky test fixes by impact
• Improved reliability: Systematic approach to CI/CD health management

Desired Outcome

This epic will be considered successful when:

• Frontend-only PRs merge in <15 minutes (vs. current 45+ minutes)
• Flaky test failure rate tracked and trending downward week-over-week
• Developers receive automated notification when failures are caused by known issues
• CI/CD health report generated weekly with actionable insights
• Zero orphaned merge queue runs consuming resources
• Circuit breaker system operational for managing catastrophic failures
• PR metrics distinguish frontend vs. backend for separate velocity tracking

Target Personas

• Developer teams
• Content teams
• DevOps teams
• System administrators (dotCMS)

Links

Related Issues

• #34592 - Cancel orphaned merge queue runs
• #34538 - ImportUtilTest flaky failures
• #33788 - Task240306MigrateLegacyLanguageVariablesTest flaky
• #34639 - Phase 2 Matrix Build Strategy

Related PRs/ADRs

• ADR-0013: Skip tests for frontend-only changes

Documentation

• RCA Document: Google Doc link to be added
• CICD_RELIABILITY_RCA.md - Technical analysis

Child Issues (To Be Created)

• Implement path-based filtering in merge queue (ADR-0013)
• Implement orphaned run cancellation (#34592)
• Create known-flaky-tests.json database
• Build failure analysis action with PR comments
• Implement circuit breaker system with GitHub variables
• Create CI/CD health metrics collection workflow
• Build weekly health report generator
• Add PR categorization (frontend/backend/mixed) to initialize phase

Start by reading docs/CICD_RELIABILITY_RCA.md and ADR-0013, then review the related issues and proposed child issues to understand the separate workstreams. This epic is done only when the listed outcomes—faster frontend merges, flaky-test tracking, failure notifications, health reporting, orphan cancellation, circuit-breaker management, and PR categorization—are addressed.