NVIDIA / NVIDIA/cudf

[BUG] Investigate the use of memory resources in Parquet stack

Open
#21,282 0 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

## Description
In the Parquet reader/writer, we may be using the `_mr` (the memory resource with which the output device memory should be allocated) to also allocate internal memory/vectors and (highly unlikely) vice versa with the `cudf::get_current_device_resource_ref()` for output memory.

## Proposed fix
Create a new member variable called `internal_mr` or `temp_mr` initialized with `cudf::get_current_device_resource_ref()` or user provided temp_mr (as per proposed libcudf API design) and use that to allocate all internal memory (strictly no use of `cudf::get_current_device_resource_ref()` or not-providing an `mr`). xref https://github.com/rapidsai/cudf/issues/20780

## Additional context
Hmm, since this memory is not for the output table, we should actually use `cudf::get_current_device_resource_ref()` for all these. That said, I think this is likely a parquet-wide issue that we need to fix (in a separate PR of course).

_Originally posted by @mhaseeb123 in https://github.com/rapidsai/cudf/pull/21139#discussion_r2748444735_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.