Feature request: pluggable StateBackend trait — durable state for ephemeral execution environments

**Note (2026-06-04):** This issue has been substantially revised following community discussion. The original framing — "persist manifest.json across ephemeral pods" — was too narrow. After research into the academic literature and the existing tooling landscape, the underlying problem appears to be a missing architectural primitive in the composable lakehouse stack: **coordinated three-way versioning of code, data, and pipeline execution state**. This issue now proposes that abstraction as a foundation for dbt Fusion. The original SlateDB-backed state persistence is one reference implementation of the broader pattern.

---

## Problem

In a composable lakehouse stack (git + Iceberg catalog + dbt), three versioning systems operate independently:

- **Code** is versioned by git
- **Data** is versioned by the catalog (Nessie commits, Iceberg snapshots)
- **Pipeline execution state** is held in a local `manifest.json` that is overwritten on every run

This decoupling creates a class of problems that no individual tool can solve:

1. **Non-reproducibility** — given a metric computed three months ago, there is no deterministic way to recover the exact code + data + pipeline state that produced it. Audit and regulatory reproducibility require this triplet.

2. **Fragile rollback** — `git revert` rolls back code, Nessie can roll back data, but the pipeline execution state remains a single overwritten file. Rolling back only one or two of the three leaves the system in an inconsistent state.

3. **No isolated experimentation** — branching code (git) and data (Nessie) is instantaneous and zero-copy, but the pipeline state cannot be branched. Multiple teams, CI runs, or AI agents cannot meaningfully work in parallel without interfering with the production state.

4. **No audit trail of pipeline execution** — each run overwrites the previous state. There is no history of what ran, when, on which data snapshot, from which code revision.

## The pattern: coordinated three-way versioning

This pattern has academic grounding. Sheng et al. (2026) — *"Building a Correct-by-Design Lakehouse: Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents"* — formalize the model and verify it with the Alloy model checker. Every pipeline run is bound by a triplet:

```
T = { σ_code, γ_catalog, ρ_pipeline }
```

- `σ_code` — git commit SHA at run time
- `γ_catalog` — catalog snapshot hash representing a consistent state across all tables simultaneously (multi-table atomic commit)
- `ρ_pipeline` — unique run identifier, historized

The pattern is also documented in bioinformatics under the name *Reproducibility Triangle* (Code + Data + Compute Environment) and implemented natively by Bauplan, Y42, and Pachyderm — none of which integrates with dbt.

## What this enables

### Concrete production scenario — insurance pricing pipeline

**Stack:** PostgreSQL operational → Bronze Iceberg (Nessie) → Silver/Gold dbt models → Power BI dashboards used by underwriters
**Execution:** Airflow on Kubernetes, ephemeral pods, daily 5am refresh

**Monday 8:30 AM** — underwriters open Power BI. Claims frequency shows +38% vs Friday. Pricing has already been quoted to brokers for the day.

**Without the triplet:**
- Manual investigation: which code changed? When? On what data?
- Reproducing Friday's numbers requires manual git checkout + manual Nessie reset + hope the local environment matches what ran in production
- Time-to-mitigation: hours

**With the triplet:**
```bash
# 3 minutes — identify what changed
dbt state inspect --date "2026-06-04T05:00"
→ git_sha: f7a3c91
→ catalog_snapshot: snap-0187 (Nessie multi-table commit)
→ run_id: run-2026-06-04T05:00

dbt state diff --from "run-2026-06-03T05:00" --to "run-2026-06-04T05:00"
→ silver_fact_claims: body modified (PR dbt-labs/dbt-core#13839 merged Sunday evening)

# Instant mitigation — reproduce Friday's numbers in isolation
dbt env branch hotfix/restore-friday --from-run "run-2026-06-03T05:00"
dbt run --env hotfix/restore-friday --select gold_claims_frequency
→ underwriters see correct numbers in Power BI
→ production main branch untouched
```

### AI agents — the strongest forward-looking case

Autonomous data agents (investigating anomalies, retraining models, proposing schema changes) cannot safely operate on production state. The triplet enables true sandboxing:

```bash
agent receives task: "investigate Q3 revenue drop"
agent creates:
git branch agent/investigate-q3-drop
nessie branch agent/investigate-q3-drop (zero-copy)
state branch agent/investigate-q3-drop (isolated history)

→ agent explores, modifies models, tests hypotheses
→ every run tracked in state@agent/...
→ production data never touched
→ agent submits PR + diff for human review with full audit trail
```

This addresses the safety requirement that emerges from EU AI Act obligations on high-risk systems: traceability of training data, model versioning, decision audit trails — answered by a single mechanism.

### Regulatory and BI reproducibility

- **Finance** — BCBS 239, IFRS 9 stress tests: any reported metric must be reproducible from the exact code + data + execution
- **Pharma** — 21 CFR Part 11 for clinical trials: full audit trail of analyses
- **BI consistency** — when a CFO asks "why are Q1 numbers different from what we presented in March", the answer is one inspect command, not a forensic investigation

## Proposed abstractions

### 1. `StateBackend` trait — historized, pluggable

```rust
#[async_trait]
pub trait StateBackend: Send + Sync {
async fn put(&self, run_id: &str, key: &str, value: Vec)
-> Result<(), StateBackendError>;
async fn get_at(&self, run_id: &str, key: &str)
-> Result>, StateBackendError>;
async fn get_latest(&self, branch: &str, key: &str)
-> Result>, StateBackendError>;
async fn list_runs(&self, branch: &str)
-> Result, StateBackendError>;
async fn set_latest(&self, branch: &str, run_id: &str)
-> Result<(), StateBackendError>;
fn describe(&self) -> String;
}
```

Two reference implementations:
- `FilesystemBackend` — current behavior preserved, no regression
- `SlateDbBackend` (optional feature flag) — embedded LSM-tree on S3/MinIO, formally verified single-writer fencing, native immutability of past states

### 2. `CatalogVersionProvider` trait — catalog-agnostic

```rust
pub trait CatalogVersionProvider: Send + Sync {
async fn current_snapshot(&self, branch: &str)
-> Result;
async fn create_branch(&self, name: &str, from: &str)
-> Result<(), CatalogError>;
fn catalog_type(&self) -> &str;
fn supports_atomic_multi_table_commit(&self) -> bool;
}

pub enum CatalogSnapshot {
/// Catalogs with native multi-table atomic commits (Nessie, lakeFS)
Atomic(String),
/// Catalogs that only track per-table snapshots (Polaris, Glue, Unity)
PerTable(HashMap),
}
```

**Important architectural note:** Nessie and lakeFS provide multi-table atomic commits — a single hash represents the consistent state of all tables. Apache Polaris, Unity Catalog, and AWS Glue (today) only track per-table snapshots, which does not provide the same consistency guarantee. This trait documents that distinction explicitly rather than hiding it.

### 3. `RunMetadata` — the triplet, captured per run

```rust
pub struct RunMetadata {
pub run_id: String,
pub timestamp: DateTime,
pub git_sha: Option,
pub catalog_snapshot: Option,
pub catalog_type: Option,
pub branch: String,
}
```

Captured at run initialization in `dbt-main/src/compilation.rs`, persisted by the `StateBackend` alongside the manifest.

### 4. `--state` flag accepts URIs

```bash
# Current behavior unchanged
dbt run --state ./target/prod

# New: durable, historized, branched
dbt run --state "slatedb://minio:9000/dbt-state@main"
```

### 5. New commands (subset; not all needed in initial PR)

```bash
dbt state inspect --run-id
dbt state diff --from --to
dbt state list-runs --branch
dbt env branch --from
dbt env diff
```

## Position in the landscape

| Approach | Code | Catalog | Pipeline state | Gap |
|---|---|---|---|---|
| **Bauplan** | Code zip per run | Iceberg + REST catalog | Native | Vendor-specific, replaces dbt |
| **Y42** | Git | DWH view pointers | Asset-based | Cloud DWH only, replaces dbt |
| **Pachyderm** | Docker images | PFS (file-based) | Global Commit ID | File-based, not table-based |
| **SQLMesh** | Git | Model fingerprinting | External state DB | No catalog-level atomic transactions |
| **dbt + Nessie + git (today)** | Git ✓ | Nessie ✓ | Local file ✗ | **This proposal** |

The "composable stack" quadrant — open source, no vendor lock-in, dbt-native — is where this proposal fits. None of the existing solutions occupy it.

## Honest acknowledgments

### Overlap with dbt State (managed product)

This proposal overlaps with the dbt State product's value proposition. We see it as complementary, not competitive:

- dbt State transmits SQL hashes to dbt Labs servers. Teams under data residency constraints, sovereign cloud requirements, or air-gapped environments cannot adopt it regardless of pricing.
- This proposal serves that segment without cannibalizing dbt Cloud customers, who benefit from managed capabilities (automatic sync, cross-environment promotion UI, support SLA) that this trait does not replace.
- The trait could itself become a substrate that dbt Cloud uses internally to offer richer guarantees in the managed product.

### Nessie's structural advantage for γ_catalog

The elegance of the triplet relies on a catalog that supports multi-table atomic commits. Today this is Nessie and lakeFS. Polaris, Unity Catalog, and Glue do not provide this — implementations of `CatalogVersionProvider` over those catalogs would fall back to per-table snapshots, which is traceable but loses the cross-table consistency guarantee. We see this as an opportunity: the trait defines the interface that catalogs *should* implement to participate fully in this pattern.

### Security

- Credentials must never appear in `--state` URIs. Resolution via env vars, IAM roles, or external credential providers only.
- TLS support is mandatory for object-store backends.
- The state contains compiled SQL — sensitive metadata that must support server-side encryption at rest in the chosen backend.

## What this does NOT change

- All existing `--state ./path` invocations work identically
- State comparison logic (SHA256 body, configs, relations) is untouched
- dbt State (managed product) is unaffected
- No new mandatory dependencies; SlateDB and catalog providers are feature-flagged

## Questions for maintainers

1. Is the broader pattern (coordinated three-way versioning) something dbt Fusion intends to support, or is the scope intentionally narrower than this proposal?
2. Would a phased contribution be welcomed — starting with `StateBackend` + historization, deferring `CatalogVersionProvider` and branch commands to follow-ups?
3. Is there an existing roadmap discussion or RFC venue more appropriate than this issue for an architectural proposal of this scope?

## References

- Sheng et al. (2026). *Building a Correct-by-Design Lakehouse: Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents.*
- The Reproducibility Triangle (bioinformatics) — Code + Data + Compute Environment as the minimal set for reproducible scientific pipelines.
- Bauplan, Y42, Pachyderm — existing native implementations of the pattern outside the dbt ecosystem.
- Proof-of-concept implementation: [egwada/dbt-fusion](https://github.com/egwada/dbt-fusion)