JuliaData / JuliaData/CSV.jl

Allow lazy materialization of iterated CSV.Chunks

Open
#970 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
506
Forks
150
Avg merge
4d 16h
Merged PRs (30d)
7

Description

While discussing out-of-core data processing with @bkamins, we realized it's currently a bit awkward to work with CSV.Chunks. For context, CSV.Chunks is currently structured to:

  • parse the header and initial data positions for each chunk in CSV.Chunks constructor
  • When CSV.Chunks is iterated, the CSV.Context + byte position is passed to CSV.File for parsing and CSV.File is returned

That means it's really difficult, if not impossible, if you wanted to spawn the chunk parsing to a separate thread/remote process, since it's baked in to the call to iterate. It would be more helpful if we structured it more similarly to Tables.partitioner(f, iter), where f(x) generates a LazyTable and the table isn't materialized until Tables.columns or Tables.rows is called. So the API for CSV.Chunks would become:

  • chunks = CSV.Chunks(...)
  • x, st = iterate(chunks)
  • file = CSV.materialize(x) where materialize takes x, which would be some kind of CSV.Chunk object that noted the CSV.Context and starting byte position, and materialize would call the actual CSV.File parsing

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the CSV.Chunks constructor and iterate implementation, then compare the proposed shape with Tables.partitioner(f, iter). Define how iteration yields a lazy CSV.Chunk carrying the CSV.Context and byte position, and how CSV.materialize(x) produces the parsed CSV.File; done means chunk parsing is deferred until materialization.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.