cockroachdb / cockroachdb/cockroach

roachprod-centralized: OpenTofu-based provisioning system

Open
#164,210 1 comment 0 reactions 1 assignee Claimed by @golgeek View on GitHub
A-testeng-infra A-testing C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

# Summary

Add a provisioning system to roachprod-centralized that enables users to define infrastructure as OpenTofu templates and provision/destroy instances via the REST API and `roachprod` CLI. This replaces the need for imperative cloud SDK code per topology and centralizes credentials, state management, and lifecycle (TTL, GC).

# Motivation

roachprod-centralized currently discovers and tracks clusters provisioned externally (via `gcloud`/`awscli`). This breaks down for complex multi-resource topologies (e.g., RDS + EC2 in the same VPC, Oracle + EC2, Spanner + GCE, ...) where each topology would require custom imperative cloud SDK code. The provisioning system lets users express infrastructure as declarative OpenTofu templates and manages the full lifecycle through the existing task system.

This work is designed to run in parallel of the existing roachprod clusters feature. The primary goal is a way to quickly provision infrastructure for competitive benchmarking, and while it is expected that the new OpenTofu provisioner will end up replacing the clusters creation part of roachprod, this is not part of the PoC.

# Key design decisions

- *OpenTofu executor via direct `os/exec` wrapper (not `tofu-exec`, which is pre-v1.0)
- *GCS backend* for OpenTofu state (CockroachDB lacks `pg_advisory_lock` support, making the `pg` backend unsafe for concurrent operations)
- *Template snapshots* stored as BYTEA in CRDB (<100KB compressed) for destroy reproducibility
- *Environments* as first-class entities with three variable types: `plaintext`, `template_secret`, `secret` — each with distinct routing to OpenTofu (`-var` flags, `TF_VAR_*` env vars, raw env vars)
- *8-char random identifier* injected into every template for resource name uniqueness
- *Task linkage vs locking* via separate `reference` and `concurrency_key` columns

# PoC work units

| # | Description |
|---|---|
| 1 | Template manager + HCL variable parser |
| 2 | Environments vertical slice (model, repo, service, controller, GCP secret resolver) |
| 3 | OpenTofu executor (init/plan/apply/destroy/output) |
| 4 | Local provisioning CLI command + variable assembly (BuildVarMaps) |
| 5 | Provisioning model + CRDB/memory repository + migrations |
| 6 | Provisioning service + task handlers + controller + factory wiring |
| 7 | Task log streaming (LogSink, ILogStore, SSE endpoint) |
| 8 | TTL GC watcher + lifetime extension + Backend interface |
| 9 | roachprod CLI (environments, provisionings, tasks) |
| 10 | Environment variable type refactor (plaintext/secret/template_secret) |
| 11 | Hooks framework after provisionings |
| 12 | SSH keys setup hook |

# API surface (after all work units)

- `POST/GET/PUT/DELETE /v1/environments` + variable sub-resource CRUD
- `POST /v1/provisionings` — create provisioning (returns task_id)
- `GET /v1/provisionings` — list with filtering (state, environment, owner)
- `GET /v1/provisionings/:id` — get provisioning
- `GET /v1/provisionings/:id/plan` — get captured plan JSON
- `GET /v1/provisionings/:id/outputs` — get terraform outputs
- `POST /v1/provisionings/:id/destroy` — trigger destroy (returns task_id)
- `DELETE /v1/provisionings/:id` — delete record (only from new/destroyed)
- `PATCH /v1/provisionings/:id/lifetime` — extend TTL
- `GET /v1/tasks/:id/logs` — SSE stream of task logs

# CLI surface

- roachprod env {list,create,delete}
- roachprod env var {list,get,set,delete}
- roachprod prov templates []
- roachprod prov create --type --env --var k=v
- roachprod prov {list,get,destroy,delete,plan,outputs,extend,logs}
- roachprod task {list,logs}

Epic CRDB-60563

Jira issue: CRDB-60689

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.