NVIDIA / NVIDIA/cudf

[QST] CPU memory spike during cudf dataframe conversion

Open
#12,501 3 comments 0 reactions 0 assignees View on GitHub
0 - Backlog Python question
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

Hi all, I have a dataframe that is ~19K Rows, ~11.4 MB (Profiled using ```df.info(memory_usage = "deep")```). We are currently running into CPU out of memory issues and so profiling our memory using this sample dataset. As you can see in the screenshot attached, there is a jump in mem usage, from 840MiB -> 4148MiB, during the type conversion of ```df```. Image below shows the dataframe memory usage after conversion.

My question is: Why is there a jump in the memory usage when converting a dataframe from pandas to cudf? Furthermore, this memory is not released after, and so increases from this point in following processing steps.

Screenshot 2023-01-09 at 6 08 11 PM
Screenshot 2023-01-10 at 11 16 09 AM

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.