Altinity / Altinity/altinity-sql-browser

Schema lineage: cap the EXPLAIN AST fan-out on large schemas

Open
#42 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
8
Forks
2
Avg merge
1h 34m
Merged PRs (30d)
6

Description

Problem

loadSchemaLineage (src/net/ch-client.js) resolves a view/MV's source tables by running EXPLAIN AST <as_select> per object, fanned out with a single unbounded Promise.all:

await Promise.all(tables.map(async (t) => {
  if (!t.as_select || (t.engine !== 'View' && t.engine !== 'MaterializedView')) return;
  try {
    const ast = await queryJson(ctx, 'EXPLAIN AST ' + t.as_select);
    t.astTables = parseAstTables((ast.data || []).map((r) => r.explain).join('\n'));
  } catch { /* best-effort */ }
}));

So the number of EXPLAIN AST queries launched simultaneously equals the number of views + materialized views in the database. There is no application-level concurrency limit — the only throttle is the browser's connection pool:

  • HTTP/1.1 — ~6 concurrent per origin (the rest queue).
  • HTTP/2 (what the TLS demo clusters serve) — all multiplexed over one connection, so effectively all fire near-simultaneously.

On a database with hundreds of views/MVs (e.g. github.demo's big schemas), dragging it onto the results pane can spray hundreds of concurrent EXPLAIN AST queries at ClickHouse in one burst.

Impact

  • Burst load on the ClickHouse server proportional to the view/MV count.
  • Slow time-to-graph for large schemas (mitigated UX-wise by the existing "Loading lineage…" state, but the work itself is still unbounded).

Proposed fix (either, or both)

  1. Bound the fan-out — run the EXPLAIN AST queries through a small worker pool (e.g. 4–8 in flight) instead of Promise.all over everything.
  2. Prefer structured columns first — when dependencies_database/dependencies_table are already populated for a view/MV, skip its EXPLAIN AST entirely and only fall back to parsing for the ones where the structured columns are empty. On modern builds this eliminates most/all of the fan-out; on older builds (e.g. Altinity-antalya 26.3, where dependencies_* is often empty) it falls back as today.

(2) is the bigger win where it applies; (1) is the robust backstop for the builds that still need the parse path.

Context

Introduced with the schema lineage graph (#41). The graph math (src/core/schema-graph.js) is unaffected — this is purely about how the loader gathers source-table data. A cap belongs in loadSchemaLineage; the structured-first short-circuit can be decided per-row there too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/net/ch-client.js at loadSchemaLineage and trace how dependencies_database/dependencies_table and EXPLAIN AST populate each table. Implement the bounded and/or structured-first behavior described; done means lineage still resolves views and materialized views while EXPLAIN AST requests no longer launch unbounded.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, javascript
Domain
databases, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.