Altinity / Altinity/altinity-sql-browser
Schema lineage: cap the EXPLAIN AST fan-out on large schemas
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- TypeScript
- Sterne
- 8
- Forks
- 2
- Ø Merge
- 1 Std. 34 Min.
- Gemergte PRs (30 T.)
- 6
Beschreibung
Problem
loadSchemaLineage (src/net/ch-client.js) resolves a view/MV's source tables by running EXPLAIN AST <as_select> per object, fanned out with a single unbounded Promise.all:
await Promise.all(tables.map(async (t) => {
if (!t.as_select || (t.engine !== 'View' && t.engine !== 'MaterializedView')) return;
try {
const ast = await queryJson(ctx, 'EXPLAIN AST ' + t.as_select);
t.astTables = parseAstTables((ast.data || []).map((r) => r.explain).join('\n'));
} catch { /* best-effort */ }
}));
So the number of EXPLAIN AST queries launched simultaneously equals the number of views + materialized views in the database. There is no application-level concurrency limit — the only throttle is the browser's connection pool:
- HTTP/1.1 — ~6 concurrent per origin (the rest queue).
- HTTP/2 (what the TLS demo clusters serve) — all multiplexed over one connection, so effectively all fire near-simultaneously.
On a database with hundreds of views/MVs (e.g. github.demo's big schemas), dragging it onto the results pane can spray hundreds of concurrent EXPLAIN AST queries at ClickHouse in one burst.
Impact
- Burst load on the ClickHouse server proportional to the view/MV count.
- Slow time-to-graph for large schemas (mitigated UX-wise by the existing "Loading lineage…" state, but the work itself is still unbounded).
Proposed fix (either, or both)
- Bound the fan-out — run the
EXPLAIN ASTqueries through a small worker pool (e.g. 4–8 in flight) instead ofPromise.allover everything. - Prefer structured columns first — when
dependencies_database/dependencies_tableare already populated for a view/MV, skip itsEXPLAIN ASTentirely and only fall back to parsing for the ones where the structured columns are empty. On modern builds this eliminates most/all of the fan-out; on older builds (e.g. Altinity-antalya 26.3, wheredependencies_*is often empty) it falls back as today.
(2) is the bigger win where it applies; (1) is the robust backstop for the builds that still need the parse path.
Context
Introduced with the schema lineage graph (#41). The graph math (src/core/schema-graph.js) is unaffected — this is purely about how the loader gathers source-table data. A cap belongs in loadSchemaLineage; the structured-first short-circuit can be decided per-row there too.
Beitragsleitfaden
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Beginne in src/net/ch-client.js bei loadSchemaLineage und verfolge, wie dependencies_database/dependencies_table und EXPLAIN AST jede Tabelle befüllen. Implementiere das beschriebene Verhalten mit Begrenzung und/oder Vorrang für strukturierte Daten; abgeschlossen ist die Arbeit, wenn die Lineage weiterhin Views und materialized views auflöst und EXPLAIN AST-Anfragen keine unbegrenzten Vorgänge mehr starten.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- clickhouse, javascript
- Bereich
- databases, performance
- Issue-Typ
- Bug
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 68/100