NVIDIA / NVIDIA/cudf

[FEA] Add support to low_memory parameter in read_csv method

Open
#4,999 13 comments 0 reactions 0 assignees View on GitHub
cuIO feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
Hi,

It would be great if you could please evaluate the addition of the `low_memory` parameter in cuDF's `read_csv` method.

This parameter defaults to `True` in Pandas' `read_csv` method, and it indicates Pandas to read the CSV file in chunks, allowing to load a large CSV file on memory-constrained systems.

Current cuDF's `read_csv` implementation first reads the whole file into GPU memory, and then creates the DataFrame. That has the benefit of a very fast DataFrame creation, but it also has the cons that it limits the size of the biggest dataset it can be created from a file.

In my tests, I was not able to create a DataFrame bigger than 7 GBs in a 16GBs v100 card.

I think that, by reading the CSV file in chunks, the max size of the DataFrame that I could create from a CSV file would be much bigger, and I also think that the potential (if any) performance penalty might be well-worth.

**Describe the solution you'd like**
To be able to load bigger datasets when reading CSV files with cuDF.

**Describe alternatives you've considered**
I have manually read the file in chunks, using the byte_range parameter, and then concatenating the different dataframes. I was able to load a DF up to 10GBs without a significate performance penalty. Also, my solution implies concat to dataframes, which would not be needed in a native libcudf implementation.

**Additional context**
Discussed internally with @kkraus14 .

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.