Altinity / Altinity/altinity-sql-browser
Schema lineage: cap the EXPLAIN AST fan-out on large schemas
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 8
- Forks
- 2
- Avg merge
- 1h 34m
- Merged PRs (30d)
- 6
Description
Problem
loadSchemaLineage (src/net/ch-client.js) resolves a view/MV's source tables by running EXPLAIN AST <as_select> per object, fanned out with a single unbounded Promise.all:
await Promise.all(tables.map(async (t) => {
if (!t.as_select || (t.engine !== 'View' && t.engine !== 'MaterializedView')) return;
try {
const ast = await queryJson(ctx, 'EXPLAIN AST ' + t.as_select);
t.astTables = parseAstTables((ast.data || []).map((r) => r.explain).join('\n'));
} catch { /* best-effort */ }
}));
So the number of EXPLAIN AST queries launched simultaneously equals the number of views + materialized views in the database. There is no application-level concurrency limit — the only throttle is the browser's connection pool:
- HTTP/1.1 — ~6 concurrent per origin (the rest queue).
- HTTP/2 (what the TLS demo clusters serve) — all multiplexed over one connection, so effectively all fire near-simultaneously.
On a database with hundreds of views/MVs (e.g. github.demo's big schemas), dragging it onto the results pane can spray hundreds of concurrent EXPLAIN AST queries at ClickHouse in one burst.
Impact
- Burst load on the ClickHouse server proportional to the view/MV count.
- Slow time-to-graph for large schemas (mitigated UX-wise by the existing "Loading lineage…" state, but the work itself is still unbounded).
Proposed fix (either, or both)
- Bound the fan-out — run the
EXPLAIN ASTqueries through a small worker pool (e.g. 4–8 in flight) instead ofPromise.allover everything. - Prefer structured columns first — when
dependencies_database/dependencies_tableare already populated for a view/MV, skip itsEXPLAIN ASTentirely and only fall back to parsing for the ones where the structured columns are empty. On modern builds this eliminates most/all of the fan-out; on older builds (e.g. Altinity-antalya 26.3, wheredependencies_*is often empty) it falls back as today.
(2) is the bigger win where it applies; (1) is the robust backstop for the builds that still need the parse path.
Context
Introduced with the schema lineage graph (#41). The graph math (src/core/schema-graph.js) is unaffected — this is purely about how the loader gathers source-table data. A cap belongs in loadSchemaLineage; the structured-first short-circuit can be decided per-row there too.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/net/ch-client.js at loadSchemaLineage and trace how dependencies_database/dependencies_table and EXPLAIN AST populate each table. Implement the bounded and/or structured-first behavior described; done means lineage still resolves views and materialized views while EXPLAIN AST requests no longer launch unbounded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, javascript
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100