Streaming reads for files larger than RAM
Open
area:data
help wanted
- Dominant language
- Rust
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
DuckDB can read Parquet (and CSV/JSON via \`read_csv_auto\`) in chunks without loading the whole file. xeli currently does a full load.
**Goal**: open a 10GB Parquet without 10GB of RAM. The table can show "page N of N" with viewport-driven row fetches.
**Where**: \`src/data/engine.rs\` (\`load\` + new \`load_lazy\`), \`src/data/loader.rs\` (decide threshold), \`src/ui/table.rs\` (page-aware rendering — already viewport-windowed, mostly plumbing).
Open a tracking issue with your proposed API before coding — there's some app-state ripple.
Contributor guide
Assessment
This issue has not been assessed yet.