Altinity / Altinity/altinity-sql-browser
Schema lineage: cap the EXPLAIN AST fan-out on large schemas
Nessuno ha ancora preso questa issue.
- Lingua principale
- TypeScript
- Stelle
- 8
- Fork
- 2
- Merge medio
- 1h 34m
- PR unite (30g)
- 6
Descrizione
Problem
loadSchemaLineage (src/net/ch-client.js) resolves a view/MV's source tables by running EXPLAIN AST <as_select> per object, fanned out with a single unbounded Promise.all:
await Promise.all(tables.map(async (t) => {
if (!t.as_select || (t.engine !== 'View' && t.engine !== 'MaterializedView')) return;
try {
const ast = await queryJson(ctx, 'EXPLAIN AST ' + t.as_select);
t.astTables = parseAstTables((ast.data || []).map((r) => r.explain).join('\n'));
} catch { /* best-effort */ }
}));
So the number of EXPLAIN AST queries launched simultaneously equals the number of views + materialized views in the database. There is no application-level concurrency limit — the only throttle is the browser's connection pool:
- HTTP/1.1 — ~6 concurrent per origin (the rest queue).
- HTTP/2 (what the TLS demo clusters serve) — all multiplexed over one connection, so effectively all fire near-simultaneously.
On a database with hundreds of views/MVs (e.g. github.demo's big schemas), dragging it onto the results pane can spray hundreds of concurrent EXPLAIN AST queries at ClickHouse in one burst.
Impact
- Burst load on the ClickHouse server proportional to the view/MV count.
- Slow time-to-graph for large schemas (mitigated UX-wise by the existing "Loading lineage…" state, but the work itself is still unbounded).
Proposed fix (either, or both)
- Bound the fan-out — run the
EXPLAIN ASTqueries through a small worker pool (e.g. 4–8 in flight) instead ofPromise.allover everything. - Prefer structured columns first — when
dependencies_database/dependencies_tableare already populated for a view/MV, skip itsEXPLAIN ASTentirely and only fall back to parsing for the ones where the structured columns are empty. On modern builds this eliminates most/all of the fan-out; on older builds (e.g. Altinity-antalya 26.3, wheredependencies_*is often empty) it falls back as today.
(2) is the bigger win where it applies; (1) is the robust backstop for the builds that still need the parse path.
Context
Introduced with the schema lineage graph (#41). The graph math (src/core/schema-graph.js) is unaffected — this is purely about how the loader gathers source-table data. A cap belongs in loadSchemaLineage; the structured-first short-circuit can be decided per-row there too.
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Inizia in src/net/ch-client.js, da loadSchemaLineage, e segui come dependencies_database/dependencies_table ed EXPLAIN AST popolano ogni tabella. Implementa il comportamento descritto, limitato e/o con priorità alle strutture; il lavoro è completato quando il lineage continua a risolvere views e materialized views, mentre le richieste EXPLAIN AST non avviano più operazioni senza limiti.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- clickhouse, javascript
- Ambito
- databases, performance
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 68/100