AltimateAI / AltimateAI/altimate-code

dbt PR review: CI experience hides the engine's proofs (undocumented artifacts, misleading lint-only banner, silent AI lane, guessed base ref)

Đang mở
#1,240 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug
Ngôn ngữ chính
TypeScript
Star
811
Fork
134
Merge trung bình
3 ngày 2 giờ
Pull request đã merge (30 ngày)
50

Mô tả

## Summary

Running the dbt PR review end to end (altimate-code 0.10.0 on `jaffle_shop_duckdb` with five injected changes) and mining the 30 most recent reviews in our own dogfood repo surfaced a cluster of CI-experience problems. None change what the engine proves; all change whether users see it.

## Problems

1. **Artifacts that decide fidelity are undocumented.** Equivalence and lineage-breakage proofs need `target/catalog.json` (`dbt docs generate`) and compiled SQL for the base ref in `target-base/compiled`. Neither appears in the usage doc quick-start, `github/review/action.yml`, or `github/review/examples/altimate-ingestion.yml`; `target-base` is mentioned only in prose. 57% of our dogfood reviews ran lint-only.
2. **"Lint-only run — no dbt manifest/warehouse was available" is shown when a manifest was present.** `summary.degraded` is `runDegraded || findings.some(f => f.degraded)`, so a single undecidable finding flips the whole-run banner (`orchestrate.ts`, `format.ts`).
3. **The AI lane fails silently.** Every failure path in `runAiReview` returns `[]` with only a log; telemetry shows 18% of CI runs call `review_ai_prompt` and never `review_ai_parse`. A default run with no credentials attempted a stale local endpoint every time; the docs say the lane is "skipped".
4. **Base ref is guessed.** `defaultBaseRef` walks `origin/main → origin/master → main → master` and ignores the event's `pull_request.base.ref`. A PR against `deployment` was diffed against `main`.
5. **One rule is half of all findings.** "new model has no uniqueness/grain test" was 427 of 853 findings across 30 dogfood PRs, one bullet per model, never acted on.
6. **Headless review telemetry has no `project_id`.** `Telemetry.setContext` is only called from the interactive session loop, so `altimate review` events cannot be grouped by repo.
7. **PR title/body never reach the AI lane in CI**, so its intent check cannot run.
8. Docs describe PII classification as "columns that newly expose sensitive data"; the lane flags any PII column present in a touched model.

## Fix (this PR)

Separate `lintOnly` from `undecidableFindings` in the envelope and render each honestly; add artifact hints with the exact commands; AI lane returns `ok | skipped | timeout | error` with a reason, rendered in the summary and recorded in `review_run` (`ai_status`, `ai_findings`, `undecidable_findings`); scale the AI timeout with prompt size; read base ref and PR title/body from `GITHUB_EVENT_PATH`; group repeated grain-test findings in the summary while keeping atomic findings; set `project_id` on the headless path; update the docs, action and example workflow to compile head and base and generate the catalog.

## Out of scope, tracked in `docs/internal/2026-09-03-dbt-pr-review-deep-dive.md`

Verdict provenance (catalog rules can currently block, contrary to the README), loading `.altimate/review.yml` from the base rather than the PR head, gate lifecycle / override wiring, inline-comment dedupe, positional equivalence false positive (core), feedback loop.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start with the end-to-end dbt PR review on jaffle_shop_duckdb, then trace summary.degraded in orchestrate.ts and format.ts and the failure paths in runAiReview. Read github/review/action.yml and github/review/examples/altimate-ingestion.yml alongside the GITHUB_EVENT_PATH path. Done means the listed CI statuses, artifact hints, metadata, telemetry fields, grouped findings, and documentation are represented honestly.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
github-actions, sql, typescript
Lĩnh vực
ci-cd, cli, data-engineering, documentation
Loại issue
Lỗi
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.