Prototype GitOps Cluster generation

Parent Issue

No response

Task

Objective

We need to prototype the process of generating a clean, GitOps-based Kubernetes (K8s) cluster that can eventually serve as the foundation for our evergreen clusters. Our current clusters contain significant technical debt due to manual configurations, making it difficult to manage and scale efficiently.

As we transition to our new evergreen platform, we aim to approach refactoring both bottom-up and top-down. Ideally, we would progress on both fronts simultaneously, though this may not always be feasible.

---

Approach

1. Bottom-Up: Automating Cluster Creation

This approach focuses on building a repeatable, automated process for provisioning clusters with all necessary configurations to support customer applications. Key principles include:

• Modularity & DRY Principles – The setup should be modular, ensuring reusability and maintainability.
• Idempotency – The provisioning process should be re-runnable without requiring manual cleanup, while also supporting a full reset when needed.
• Hierarchical & Composable Configuration – Implementing convention over configuration to define system-wide defaults with environment- and instance-specific overrides.
• Best Practices Compliance – Ensuring the cluster setup aligns with industry best practices for security, scalability, and maintainability.

2. Top-Down: Modularizing Existing Deployments

In this approach, we focus on incrementally refactoring existing customer configurations by introducing modularity and standardization:

• Encapsulating Configurations – Utilizing Helm to package customer configurations into reusable, version-controlled charts.
• Progressive Transition – Replacing manually defined Kubernetes resources with modular, reusable components without disrupting existing deployments.

---

Goal of this Spike

The primary goal of this spike is to evaluate how far we can progress with the bottom-up approach and assess the effort required to go from scratch to a fully operational customer-ready cluster.

• Scope: Build a prototype GitOps-driven cluster provisioning process.
• Tooling: Leverage AI (e.g., Cursor) to assist in generating scripts and configurations while ensuring adherence to best practices.
• Outcome: Identify challenges, refine our approach, and determine feasibility for scaling.

By utilizing AI to assist with script and configuration generation, we aim to accelerate development and reduce complexity, making what initially seemed like an overwhelming task more manageable.

Proposed Objective

Cloud Engineering

Proposed Priority

Priority 2 - Important

Acceptance Criteria

No response

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

The issue names no files, tests, or existing entry points to inspect. Start by locating the repository's current Kubernetes and Helm deployment configuration, then define the prototype's scope and provisioning entry point. Done means a repeatable GitOps-based cluster provisioning prototype exists and its challenges and feasibility are documented.