h2oai / h2oai/datatable

fread - OOM issue

Open
#3,475 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

**Issue:**

+1 for the awesome package.

Encountering Out Of Memory issue when reading a LARGE csv file (~200 million rows) using fread.
Datatable code is being run on an **_aks-cluster_** with 20Gi of RAM and its still tripping

**Code:**
image
csv file:
image

**Expected behavior:**
Being able to load the file and perform manipulations.

**Environment:**
datatable version - 1.0.0
python version - 3.8.0
operating system - linux

Any pointers on how to do the aforementioned task in a memory efficient way would be of great help.

Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.