JuliaData / JuliaData/CSV.jl

Memory Consumption of CSV.Rows with ZipFile

Open
#997 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Julia
Stars
506
Forks
150
Avg merge
4d 16h
Merged PRs (30d)
7

Description

Hi, I am trying to use CSV.Rows to iterate through a zipped text file which is about 3.4GB uncompressed. It seems CSV.Rows is hold a large chunk of memory (about double the size of the file), which defeats the purpose.
If I load the unzipped text file then I don't see this problem. It seems this line of the code in utils.jl buffer_to_tempfile function allocated memory which isn't freed. I tried to set stream and output to nothing, and the program is holding memory about the size of the file (instead of double).

I am not sure if this is a problem with how I am using ZipFile, CSV.

using CSV, ZipFile;
z=ZipFile.Reader("test.zip")
r=z.files[1]
a= CSV.Rows(r; types=[Int64, String, Float64, Float64, Int8, Float64, Float64, Int64])
for c in a
    # custom aggregation code on c
end

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/utils.jl at buffer_to_tempfile, then reproduce the issue using CSV.Rows with a ZipFile.Reader entry as shown. Compare memory consumption with zipped and unzipped input, and consider the issue done when the zipped iteration no longer retains the unnecessary file-sized allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.