AltimateAI / AltimateAI/altimate-code

[Feature] Local column-level lineage via sqlglot (no API credentials required)

Offen
#62 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement priority:high sql-engine
Vorherrschende Sprache
TypeScript
Sterne
811
Forks
134
Ø Merge
3 T. 2 Std.
Gemergte PRs (30 T.)
50

Beschreibung

## Problem

The `lineage_check` tool requires AltimateCore API credentials that aren't part of the standard setup. This means column-level lineage — one of the most differentiated features — doesn't work out of the box.

Users see: "requires AltimateCore API credentials" with no guidance on how to configure them or any local alternative.

## Desired Behavior

Column-level lineage should work immediately using local analysis, with the API-based lineage available as an optional enhancement for advanced use cases.

## Implementation Notes

### Key Insight
Every major open-source lineage tool (SQLMesh, DataHub, Recce, dbt-column-lineage) uses **sqlglot's built-in `lineage()` function** for local column-level lineage. No API credentials needed.

### Key Files
- The tool that currently calls the API for lineage
- `packages/altimate-engine/` — Python engine where sqlglot lineage can be implemented
- Schema cache (SQLite) — already stores table/column metadata that can feed into lineage

### Approach

**Phase 1: Local lineage via sqlglot**
- Add a Python bridge method `lineage_local(sql, column, dialect, schema?)` that wraps sqlglot's `lineage()` function
- Feed schema information from the schema cache into the lineage function for accuracy
- Return a column dependency graph (source_table.column → target_table.column)

**Phase 2: Schema-aware lineage**
- When schema cache is populated, automatically provide table schemas to sqlglot for resolving `SELECT *` and unqualified columns
- Without schema, degrade gracefully (skip `SELECT *` expansion, note the limitation)

**Phase 3: API as premium enhancement**
- Keep API-based lineage for cross-project lineage, confidence scoring, and historical tracking
- Use local lineage as the default, API lineage as opt-in upgrade
- When API credentials aren't configured, use local lineage silently (no error)

### sqlglot lineage API
```python
from sqlglot.lineage import lineage

# Basic usage
node = lineage(
column="col_name",
sql="SELECT a AS col_name FROM table_b",
schema={"table_b": {"a": "int", "b": "varchar"}},
dialect="snowflake"
)
# node.source, node.expression, node.downstream
```

### Industry Patterns
- **SQLMesh**: Wraps sqlglot lineage with schema from warehouse introspection, provides interactive UI
- **DataHub**: sqlglot fork achieving 97-99% accuracy by combining with metadata registry
- **Recce**: Parses SQL → traverses scopes → traces columns through CTEs, classifies transformations

## Acceptance Criteria

- [ ] `lineage_check` works without any API credentials configured
- [ ] Local lineage correctly traces columns through JOINs, CTEs, subqueries, and aliases
- [ ] Schema cache data is used when available for `SELECT *` expansion
- [ ] Graceful degradation when schema is unavailable (partial lineage with notes)
- [ ] API-based lineage still works when credentials are configured (as enhancement, not requirement)

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.