Altinity / Altinity/altinity-sql-browser

GS: Failure-safe full refresh and large-result transfer

Open
#402 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cool feature
Dominant language
TypeScript
Stars
8
Forks
2
Avg merge
1h 34m
Merged PRs (30d)
6

Description

Parent: #398
Depends on: #399, #401

Design:

  • docs/design/google-sheets/full-refresh.md
  • docs/design/google-sheets/large-results.md
  • docs/design/google-sheets/failure-recovery.md

Scope

Implement manual full refresh with streaming ClickHouse reads, canonical value conversion, hidden staging, stable-sheet publication, limits, progress, cancellation, and failure recovery.

Acceptance criteria

  • The complete query result is never required in browser memory.
  • Google writes use byte-bounded explicit ranges and RAW semantics.
  • Existing live data remains unchanged until all query and staging work succeeds.
  • Final publication preserves the managed numeric sheet ID and atomically updates state.
  • Old trailing rows/columns are cleared and formatting/metadata are refreshed.
  • Ambiguous publication is reconciled by generation/refresh ID before retry.
  • Row, cell, byte, duration, and spreadsheet-capacity limits are enforced without truncation.
  • Cancellation and closed-tab recovery leave committed data intact and clean orphan staging later.
  • Successful full refresh can initialize/reset an append checkpoint when eligible.

Non-goals

Append page selection, automatic scheduling, or in-place partial replacement when staging cannot fit.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read docs/design/google-sheets/full-refresh.md, docs/design/google-sheets/large-results.md, and docs/design/google-sheets/failure-recovery.md, then review prerequisites #399 and #401. Done means implementing the listed refresh, staging, publication, limits, cancellation, recovery, and checkpoint acceptance criteria without truncation or changing committed live data prematurely.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, typescript
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.