Evergreen: Top level Epic

WIP:

Placeholder for other epics and tasks related to evergreen and how they fit together to the overall goals

```[tasklist]

Testing what we've got today

• dotCMS/core#30827
• https://github.com/dotCMS/core/issues/30909
  \`

Cluster Deployment

• [x] dotCMS/core#30645
• [ ] dotCMS/core#31157

deprioritized for now

• [ ] dotCMS/helm-charts#15

PR from "Push" to "Deployment" Flow ("core" Github Flow)

• [ ] dotCMS/core#27842

• [ ] dotCMS/core#30655
  Enable marking a particular release/docker image as having an issue and whether it is not recommended to upgrade to or recommended to downgrade from if already there.
  Enable marking a priority on a release e.g. for security issues.

• [ ] Create flow charts showing phases of deployment including stages where validations are applied and possible exception handling when an issue is found. Include where batching of PRs is included and take into account Feature Flags allow unfinished features moving into released code.

Traceability

Easily identify what images a PR was included in, and in relation which images
Easily identify what git commit hash on main and release branches any github image relates to. Can provide info in running app, could add a --version option in github startup and add info in console on startup.

Create a changelog for use in above using conventional commits
Could provide initial auto created changelog, and then allow for manual curation after the fact to prevent documentation from blocking release. Improving automation going forward (AI update of changelog?)

Testability

Define generic smoke test suite that can be run on any implementation / version to ensure it is working as expected. Can we add a version range that a test can apply to, and change tests based upon enabled flags on the server. E.g. Only test a feature if the customer is using it. Can we provide tests for when a feature is enabled or disabled. Tests should be non-evasive and not themselves break the server and should always be cleaned up. Start with simple cases.

Dry runs with customer data to find issues / gaps

Attempt to upgrade directly from LTS releases, both basic with starter and actual customer data, to local dev build and identify issues requiring manual intervention. Make use of manual upgrade steps and procedures that would be made to fix for the customer. From the issues identify what automatic interventions could have been made to prevent the issue.

Identify impact, complexity and scope of PRs to help feed into testing requirements and any version incompatibilities

Add tagging based upon file changes and use filters to match, focus on changes that could cause incompatibility in Data whether in database, search server or shared filesystem e.g. assets, push publishing data structures. Examples could be "REST API", "DB Schema", "Push Publishing", "Configuration",

Observability and Error Remediation

Cleanup logs, review production logs and identify over logging and Exceptions that do not indicate an issue that needs to be resolved. Update logging default levels or modify code with correct level. Define sets of logging levels that can be turned up to diagnose errors in particular areas when required.

Define broad areas where temporary issues can be expected and define rules on how to handle them, make use of System State service. Top two examples could be issues connecting to db or search server. In first case it must be handled cleanly if the servers are restarted or there is a temporary connection issue.

• [ ] dotCMS/core#29460

Modularity

Push for use of CDI and java interfaces to break dependencies between features and make it easier to turn on and off individual feature with minimal code changes. Allows also for easy backport of features to LTS versions and moves towards a "domain-driven-design" way of thinking.

Processes for handling urgent issues

Metrics for PR flow

K8S Environment services

Support for full named data snapshot/restore/clone within our k8s
Try to remove limitations of time and data transfer/duplication costs (zfs based?)
Ensure we can pull up a consistent state with both filesystem data and db.
Can we easily through APIs revert to a named state.
Can we easily create a snapshot state before and after upgrade.
Can we store metadata with a state that identifies the docker image version that it relates to

Upgrade Scheduling: Each cluster could have its own configuration defining rules for when the actual upgrade and modification of the docker image will occur. It should be possible for these options to be set within the UI if we allow for that instance. e.g. upgrade asap, upgrade on a particular day, within a window of time.
Initially no UI component

Process to update the image assigned to the instance.
We should move to assign an instance to a specific image sha and update that rather than relying on a changeable docker tag for repeatability and change managment as well as supporting downgrade.

No source files, tests, or single entry point are named. Start by reviewing the linked issues and the sections for deployment, traceability, testability, observability, and Kubernetes services, then split one concrete outcome into a scoped child issue with explicit completion criteria.