AltimateAI / AltimateAI/altimate-code

dbt PR review: CI experience hides the engine's proofs (undocumented artifacts, misleading lint-only banner, silent AI lane, guessed base ref)

Open
#1,240 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
811
Forks
134
Avg merge
3d 2h
Merged PRs (30d)
50

Description

## Summary

Running the dbt PR review end to end (altimate-code 0.10.0 on `jaffle_shop_duckdb` with five injected changes) and mining the 30 most recent reviews in our own dogfood repo surfaced a cluster of CI-experience problems. None change what the engine proves; all change whether users see it.

## Problems

1. **Artifacts that decide fidelity are undocumented.** Equivalence and lineage-breakage proofs need `target/catalog.json` (`dbt docs generate`) and compiled SQL for the base ref in `target-base/compiled`. Neither appears in the usage doc quick-start, `github/review/action.yml`, or `github/review/examples/altimate-ingestion.yml`; `target-base` is mentioned only in prose. 57% of our dogfood reviews ran lint-only.
2. **"Lint-only run — no dbt manifest/warehouse was available" is shown when a manifest was present.** `summary.degraded` is `runDegraded || findings.some(f => f.degraded)`, so a single undecidable finding flips the whole-run banner (`orchestrate.ts`, `format.ts`).
3. **The AI lane fails silently.** Every failure path in `runAiReview` returns `[]` with only a log; telemetry shows 18% of CI runs call `review_ai_prompt` and never `review_ai_parse`. A default run with no credentials attempted a stale local endpoint every time; the docs say the lane is "skipped".
4. **Base ref is guessed.** `defaultBaseRef` walks `origin/main → origin/master → main → master` and ignores the event's `pull_request.base.ref`. A PR against `deployment` was diffed against `main`.
5. **One rule is half of all findings.** "new model has no uniqueness/grain test" was 427 of 853 findings across 30 dogfood PRs, one bullet per model, never acted on.
6. **Headless review telemetry has no `project_id`.** `Telemetry.setContext` is only called from the interactive session loop, so `altimate review` events cannot be grouped by repo.
7. **PR title/body never reach the AI lane in CI**, so its intent check cannot run.
8. Docs describe PII classification as "columns that newly expose sensitive data"; the lane flags any PII column present in a touched model.

## Fix (this PR)

Separate `lintOnly` from `undecidableFindings` in the envelope and render each honestly; add artifact hints with the exact commands; AI lane returns `ok | skipped | timeout | error` with a reason, rendered in the summary and recorded in `review_run` (`ai_status`, `ai_findings`, `undecidable_findings`); scale the AI timeout with prompt size; read base ref and PR title/body from `GITHUB_EVENT_PATH`; group repeated grain-test findings in the summary while keeping atomic findings; set `project_id` on the headless path; update the docs, action and example workflow to compile head and base and generate the catalog.

## Out of scope, tracked in `docs/internal/2026-09-03-dbt-pr-review-deep-dive.md`

Verdict provenance (catalog rules can currently block, contrary to the README), loading `.altimate/review.yml` from the base rather than the PR head, gate lifecycle / override wiring, inline-comment dedupe, positional equivalence false positive (core), feedback loop.

Contributor guide

Open the contributing guide

Research direction

Start with the end-to-end dbt PR review on jaffle_shop_duckdb, then trace summary.degraded in orchestrate.ts and format.ts and the failure paths in runAiReview. Read github/review/action.yml and github/review/examples/altimate-ingestion.yml alongside the GITHUB_EVENT_PATH path. Done means the listed CI statuses, artifact hints, metadata, telemetry fields, grouped findings, and documentation are represented honestly.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, sql, typescript
Domain
ci-cd, cli, data-engineering, documentation
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.