RFC: Standalone CI Engine with AI Orchestration

**Status:** Draft
**Author:** Max Kainov
**Date:** 2026-04-03

## Background

Praktika runs on top of GitHub Actions as the execution engine. ClickHouse/ClickHouse#100291 proposed replacing it with a standalone engine based on independent runners polling an S3 job queue — no central process.

That design is clean but has a structural limitation: without a central process, there is no place for an AI agent to observe the pipeline and intervene mid-run. This RFC proposes an alternative with a central orchestrator VM that enables AI orchestration as a first-class capability and eliminates the GitHub Actions dependency. Supersedes ClickHouse/ClickHouse#100291.

## Problem

Classic CI at the scale of ClickHouse has the following problems:

- Trivial failures (formatting, minor bugs) require human attention, a new commit, and a full re-run — for something an LLM could fix in seconds.
- Heavy jobs (cross-platform builds, performance tests) run unconditionally even when a change is a comment edit.
- Infrastructure errors (runner crashes, flaky checkouts) are difficult to triage and retry automatically.
- No mechanism to stop early when the root cause is already clear from the first failures.
- Static job ordering ignores the change type and results from previous runs on the same PR.
- GitHub Actions itself is unreliable infrastructure with a significant maintenance burden (self-hosted runner fleet, flaky checkouts) and is hard to extend beyond its standard execution model.

## Proposal

Build a standalone Praktika CI engine with an LLM agent at the center of orchestration. A central orchestrator VM launches EC2 workers directly — replacing GitHub Actions — while the agent makes real-time decisions throughout the pipeline run.

Many problems found by CI can be resolved by an LLM without distracting the engineer: trivial bugs and formatting issues can be fixed and pushed as a new commit, infrastructure errors can be retried automatically, found issues can be triaged — fixed if straightforward, or a solution proposed if not. CI becomes a system that actively resolves problems instead of just reporting them.

The classic pipeline configuration is the *starting plan*, which the agent may adapt within hard limits encoded in that configuration (e.g. a test cannot run before its build).

## Core Design Principle

Clear boundary between deterministic and non-deterministic behavior.

**Praktika** owns correctness: job definitions, dependency ordering, artifact contracts, skip rules. Encoded in `Workflow.Config` and `Job.Config`, enforced by the Pipeline API.

**The LLM agent** owns intelligence: scheduling, prioritization, early cancellation, adaptive skipping, source edits, result interpretation. Operates freely within Praktika's constraints.

The **Pipeline API** is the interface between the two. Validates every action against the config and logs all events.

## Pipeline Orchestration

- **Orchestration loop** — drives execution: event loop, completion checks.
- **`Pipeline` API** — all side effects on jobs and workers.
- **`Agent`** (LLM only) — calls Pipeline methods directly as tools. The orchestration loop is the LLM SDK's tool-use loop.

Classic mode: a Python loop calls Pipeline methods with deterministic logic. LLM mode: the model calls the same methods as tools.

```
Classic LLM
────────────────────── ──────────────────────────────
Orchestration loop Orchestration loop
│ │ │
Pipeline API Pipeline API Agent
(LLM)
```

**Pipeline API.** `Pipeline` is a new Praktika class for reading state, mutating the plan, and controlling execution.

| Method | Description |
|--------|-------------|
| `get_plan()` | Current pipeline plan: jobs, dependencies, runner types, skip rules |
| `get_ready()` | Jobs whose dependencies are met and not yet started |
| `get_pending()` | Jobs still running or waiting for dependencies |
| `get_result(job)` | Structured `Result` of a completed job |
| `run_job(job, when=None)` | Launch a worker; `when` defers launch to a future time |
| `wait_any()` | Block until any job completes; timed-out jobs return with corresponding `Result.status` |
| `skip_job(job, reason)` | Mark as skipped; verifies no hard dependents break |
| `kill_all()` | Terminate all running and cancel all planned jobs |
| `update_plan(new_plan)` | Replace execution plan; validates against hard constraints |

**Classic orchestration loop (deterministic baseline).** First thing to implement: replicates GitHub Actions execution flow. Migration path and permanent fallback.

```python
while True:
for job in Pipeline.get_ready():
Pipeline.run_job(job)
Pipeline.wait_any()
if not Pipeline.get_pending():
break
```

More advanced deterministic loops (e.g. reordering by historical duration) can be built on the same API without an LLM.

**LLM orchestration loop (adaptive).** Pipeline API methods are registered as tools with the LLM SDK. The model calls them directly in a standard tool-use loop:

```python
pipeline = Pipeline(pr=pr, sha=sha)

response = llm.create(
system=SYSTEM_PROMPT,
tools=pipeline.as_tools(),
context={"pr": pr, "diff": diff, "previous_runs": previous_runs}
)

while response.has_tool_calls():
results = execute_tool_calls(response, pipeline)
response = llm.continue_with(results)
```

The model decides everything: skip, reorder, stop, fix. All decisions are logged into each job's `Result.info`.

## Architecture

**Pipeline dispatch.** GitHub sends a webhook (push, PR event) to a Lambda function. Lambda validates the event (author permissions, PR state, etc.) and enqueues a pipeline task into SQS. An orchestrator VM from a pre-warmed fleet picks up the task and runs the pipeline to completion.

```
GitHub webhook → Lambda → SQS → Orchestrator VM (from fleet)
```

Orchestrator VMs are small (no builds run on them) and cheap to keep warm. A fleet of ready VMs ensures instant pipeline start with no cold-start delay. Each VM handles one pipeline run at a time.

**Components.**

```
┌─────────────────────────────────────────────────────────┐
│ Central Orchestrator VM │
│ │
│ ┌──────────────┐ Pipeline API ┌─────────────────┐ │
│ │ LLM Agent │◄───────────────►│ Pipeline Impl │ │
│ │ │ │ │ │
│ └──────┬───────┘ └────────┬────────┘ │
│ │ │ │
│ ┌──────▼──────────────────────────────────▼────────┐ │
│ │ Artifact / Repo Storage │ │
│ │ (git worktree + results + S3 cache) │ │
│ └───────────────────────────────────────────────────┘ │
└──────────────────────────┬──────────────────────────────┘
│ boto3
┌────────────────┼────────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ EC2 Job │ │ EC2 Job │ │ EC2 Job │
│ Runner │ │ Runner │ │ Runner │
└──────────┘ └──────────┘ └──────────┘
```

**Central Orchestrator VM** — persistent EC2 instance for the duration of a run. Hosts agent, Pipeline implementation, git worktree, artifact storage. Pipeline API methods are passed as tools to the LLM SDK.

**Job Runners** — launched on demand from a base AMI. Run `praktika run JOB_NAME` autonomously. Health monitored via heartbeat; stuck instances terminated.

## Workers and Execution

**Worker launch.** Workers launched with EC2 user data containing the startup script:

```bash
# user data — generated per job
aws s3 cp s3://ci-bucket/runs/{run_id}/repo.tar.gz /tmp/repo.tar.gz
tar xzf /tmp/repo.tar.gz -C /home/ci
cd /home/ci/repo
praktika run JOB_NAME
```

**Repo distribution.** Workers do not clone from GitHub. The orchestrator uploads S3 archives. Each job declares which tier it needs via `Job.Config`:

| Tier | Contents | Used by |
|------|----------|---------|
| `shallow` | Source tree, no submodules | Style check, linting, test jobs |
| `shallow-submodules` | Source tree + submodules | Build jobs |
| `full` | Full history + submodules | Release tooling (on demand) |

Only required tiers are uploaded.

**Job registry.** The Pipeline implementation tracks all workers:

```
job_name → WorkerState {
instance_id, status, start_time,
last_heartbeat, result_path, s3_repo_archive, error
}
```

Status: `PENDING → SCHEDULED → RUNNING → SUCCESS | FAILED | TIMEOUT | ERROR`

EC2 lifecycle managed by Pipeline implementation via boto3. Agent never touches EC2 directly.

**Heartbeat and failure detection.** Workers write heartbeat every N seconds. Stale jobs transition to `TIMEOUT` and the instance is terminated. All failure modes converge on `FAILED`/`TIMEOUT`.

Infrastructure failures (runner crash, spot interruption) are automatically rescheduled — unlike GitHub Actions where they require manual intervention.

## Agent Capabilities

**1. Auto-fix.** Failed job → agent identifies root cause → kills remaining jobs → produces fix → pushes new commit with failure info in commit message and `Result`. CI restarts; Praktika cache reuses unaffected artifacts. On the new run, agent may prioritize the previously failed job to validate the fix early.

**2. Code review triage.** AI review job produces structured findings. Agent auto-fixes trivial ones, posts non-trivial as PR comments.

**3. Adaptive job skipping.** Agent inspects diff at start. Comment-only change → skip heavy jobs. Reasoning in `Result.info`.

**4. Early cancellation.** No-go failure detected → kill all remaining jobs immediately.

**5. Dynamic reordering.** Reorder based on change type and previous run results.

**6. Resource-saving scheduling.** Defer jobs as late as possible without increasing total duration. New push → maximum jobs can be canceled without waste.

## Safety Model

- **Auto-fix is opt-in.** Enabled per PR via label or flag. Disabled by default.
- **Agent can only push to the PR branch** it was triggered from. Push credentials are scoped to that branch.
- **Agent commits are clearly marked** (bot author, commit message references the failed job and root cause) so they are easy to audit and revert.
- **All agent actions are logged** — every tool call, decision, and outcome is recorded and visible in the CI report.
- **Allowed fix scope:** formatting, style, trivial test fixes. The agent does not attempt architectural changes, security-sensitive code, or changes outside the PR's diff scope.

## Orchestrator Failure Model

The orchestrator VM is a single point of coordination, not a single point of failure:

- **Running workers are unaffected.** Jobs run autonomously via the user data script. If the orchestrator dies, all in-flight jobs complete normally and write their `Result` to S3.
- **Pipeline state is persisted to S3** after every state change. A new orchestrator VM can pick up the run from the last persisted state and continue — no work is duplicated.
- **Watchdog.** An external process (Lambda or a peer VM) monitors the orchestrator. If it becomes unresponsive, the watchdog either restarts it or falls back to the classic deterministic loop to finish the remaining jobs.

## Resource Protection

- **EC2 instance time limit.** Every launched instance has a hard termination deadline regardless of job state. Prevents leaked instances.
- **Watchdog.** Background process monitors all running instances. Terminates any instance that misses heartbeat or exceeds its time limit.
- **EC2 cap.** Hard limit on total concurrent instances enforced before every launch, independent of agent decisions.
- **Global kill switch.** SSM parameter or S3 flag checked before every instance launch. Setting it pauses all new launches immediately.
- **Repeated failure circuit breaker.** If multiple consecutive pipeline runs fail on the same root cause, the system pauses new runs and alerts — prevents burning resources on a known broken state.

## Migration Plan

1. **Implement classic orchestration loop.** `Pipeline` class with core methods, deterministic loop. Replicate GitHub Actions execution behavior. Migrate PR and Master CI pipelines off GitHub Actions.
2. **Enable LLM mode for PRs on request.** Developer opts in (label or flag). Pipeline API methods registered as LLM tools. Classic mode remains the default and fallback.
3. **Migrate remaining pipelines** (release, backport, etc.) off GitHub Actions.