Altinity / Altinity/altinity-sql-browser

Support safe live-tail results for SELECT … STREAM

Aperta
#59 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

enhancement
Lingua principale
TypeScript
Stelle
8
Fork
2
Merge medio
1h 34m
PR unite (30g)
6

Descrizione

Context

ClickHouse 26.6 adds experimental continuous queries with SELECT … STREAM.
The browser already has useful building blocks for this:

  • HTTP response streaming via resp.body.getReader()
  • line-by-line JSONStringsEachRowWithProgress processing
  • no wait_end_of_query in the streaming result path
  • Stop already aborts the fetch and kills the query by query_id
  • mid-stream exceptions are already surfaced from in-band {"exception"} rows

This issue is about making SELECT … STREAM safe and understandable in the browser.

It is not about enabling ClickHouse streaming settings. The browser should not inject
enable_streaming_queries or table settings. Those remain user/session/server choices.

Problem

A streaming query is open-ended. The current UI/result model mostly assumes a bounded query that eventually finishes.

Main gaps:

  1. result.rows.push(...) grows memory without bound.
  2. Every chunk can trigger a full table rebuild, which becomes janky over long streams.
  3. The lifecycle reads like a normal query: “Running…”, then success/history. A stream should read as live, and Stop is the normal terminus.
  4. The user needs a tailing experience: newest rows visible by default, with a way to pause auto-follow when scrolling up.

Proposed work

  • Add detectStreaming(sql) as a pure core helper.
    • Detect STREAM as a table-expression modifier.
    • Avoid naive matching inside strings/comments where practical.
    • Prefer conservative detection; false negatives are safer than false positives.
  • Add a bounded row buffer.
    • Keep last N rows, for example 10k.
    • Track total received rows.
    • Show “showing last N of M rows” when older rows are dropped.
  • Throttle rendering.
    • Batch incoming chunks.
    • Render at a fixed cadence, for example around 10 fps.
    • Keep the throttle decision testable in core; keep rAF wiring in UI.
  • Add table tail behavior.
    • Auto-scroll to newest rows while the user is already at the bottom.
    • Pause auto-follow when the user scrolls up.
    • Show “Jump to latest” when paused.
  • Add live lifecycle UI.
    • Label state as “Streaming…” or “Live”.
    • Make Stop the primary action.
    • Show received row count and rows/sec.
    • Treat user Stop as a normal stream termination, not an error.
  • Improve capability errors.
    • If the server rejects streaming as disabled/unsupported, show a friendly hint:
      ClickHouse streaming queries require a compatible server and user-enabled
      enable_streaming_queries.

Acceptance criteria

  • A SELECT … STREAM query can run without unbounded memory growth.
  • Long-running streams do not rebuild the full table on every chunk.
  • The UI clearly distinguishes live streams from finite queries.
  • Stop cleanly terminates the stream and does not look like a failure.
  • The table follows the newest rows by default and supports pause/jump-to-latest.
  • A finite STREAM LIMIT query can complete normally.
  • Open streams are not automatically recorded as successful finite history entries.
  • Core behavior has unit coverage: streaming detection, ring buffer, received/dropped counters, and render-throttle decisions.

Files likely touched

File Change
src/core/format.js detectStreaming(sql)
src/core/stream.js bounded buffer, received counter, dropped counter, flush decision
src/ui/app.js live-mode branch, Stop lifecycle, throttled chunk handling
src/ui/results.js tail auto-scroll, pause, jump-to-latest, live readout
src/styles.css live state styling
tests/unit/* detection, buffer, throttle tests
README document user-controlled ClickHouse streaming setup

Open questions

  1. Should live mode be auto-detected only, manually toggled, or both?
  2. What should the default tail window be: 5k, 10k, or configurable?
  3. Should STREAM LIMIT be recorded to history once it completes?
  4. Should Stop on a stream be recorded anywhere, or treated as ephemeral?
  5. How strict should detectStreaming(sql) be before we need a real parser?

Non-goals

  • Enabling ClickHouse settings on behalf of the user.
  • Table DDL changes for cursor/minmax optimization.
  • Live chart animation or aggregation. That is tracked separately.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia con src/core/format.js e src/core/stream.js, quindi segui il ciclo di vita dei risultati attraverso src/ui/app.js e src/ui/results.js. Esegui i test pertinenti in tests/unit/* prima di estendere la copertura per rilevamento, buffering, contatori e throttling. Il lavoro è completo quando sono presenti risultati live limitati, un ciclo di vita dello streaming chiaro e controlli tail chiari, un comportamento pulito di Stop, il completamento degli stream finiti e indicazioni aggiornate nel README.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
sql, typescript
Ambito
documentation, frontend, testing
Tipo di issue
Funzionalità
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.