[QST] CPU memory spike during cudf dataframe conversion
Open
0 - Backlog
Python
question
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
Hi all, I have a dataframe that is ~19K Rows, ~11.4 MB (Profiled using ```df.info(memory_usage = "deep")```). We are currently running into CPU out of memory issues and so profiling our memory using this sample dataset. As you can see in the screenshot attached, there is a jump in mem usage, from 840MiB -> 4148MiB, during the type conversion of ```df```. Image below shows the dataframe memory usage after conversion.
My question is: Why is there a jump in the memory usage when converting a dataframe from pandas to cudf? Furthermore, this memory is not released after, and so increases from this point in following processing steps.
Contributor guide
Assessment
This issue has not been assessed yet.