AltimateAI / AltimateAI/altimate-code
[Feature] Local column-level lineage via sqlglot (no API credentials required)
- 主要言語
- TypeScript
- スター
- 811
- フォーク
- 134
- 平均マージ
- 3日 2時間
- マージ済み PR(30日)
- 50
説明
## Problem
The `lineage_check` tool requires AltimateCore API credentials that aren't part of the standard setup. This means column-level lineage — one of the most differentiated features — doesn't work out of the box.
Users see: "requires AltimateCore API credentials" with no guidance on how to configure them or any local alternative.
## Desired Behavior
Column-level lineage should work immediately using local analysis, with the API-based lineage available as an optional enhancement for advanced use cases.
## Implementation Notes
### Key Insight
Every major open-source lineage tool (SQLMesh, DataHub, Recce, dbt-column-lineage) uses **sqlglot's built-in `lineage()` function** for local column-level lineage. No API credentials needed.
### Key Files
- The tool that currently calls the API for lineage
- `packages/altimate-engine/` — Python engine where sqlglot lineage can be implemented
- Schema cache (SQLite) — already stores table/column metadata that can feed into lineage
### Approach
**Phase 1: Local lineage via sqlglot**
- Add a Python bridge method `lineage_local(sql, column, dialect, schema?)` that wraps sqlglot's `lineage()` function
- Feed schema information from the schema cache into the lineage function for accuracy
- Return a column dependency graph (source_table.column → target_table.column)
**Phase 2: Schema-aware lineage**
- When schema cache is populated, automatically provide table schemas to sqlglot for resolving `SELECT *` and unqualified columns
- Without schema, degrade gracefully (skip `SELECT *` expansion, note the limitation)
**Phase 3: API as premium enhancement**
- Keep API-based lineage for cross-project lineage, confidence scoring, and historical tracking
- Use local lineage as the default, API lineage as opt-in upgrade
- When API credentials aren't configured, use local lineage silently (no error)
### sqlglot lineage API
```python
from sqlglot.lineage import lineage
# Basic usage
node = lineage(
column="col_name",
sql="SELECT a AS col_name FROM table_b",
schema={"table_b": {"a": "int", "b": "varchar"}},
dialect="snowflake"
)
# node.source, node.expression, node.downstream
```
### Industry Patterns
- **SQLMesh**: Wraps sqlglot lineage with schema from warehouse introspection, provides interactive UI
- **DataHub**: sqlglot fork achieving 97-99% accuracy by combining with metadata registry
- **Recce**: Parses SQL → traverses scopes → traces columns through CTEs, classifies transformations
## Acceptance Criteria
- [ ] `lineage_check` works without any API credentials configured
- [ ] Local lineage correctly traces columns through JOINs, CTEs, subqueries, and aliases
- [ ] Schema cache data is used when available for `SELECT *` expansion
- [ ] Graceful degradation when schema is unavailable (partial lineage with notes)
- [ ] API-based lineage still works when credentials are configured (as enhancement, not requirement)
コントリビューションガイド
評価
この issue はまだ評価されていません。