NVIDIA / NVIDIA/cudf

[FEA][JNI] Add an API that allows us to build batches on the GPU from host columns

Open
#12,889 0 comments 0 reactions 0 assignees View on GitHub
0 - Backlog feature request Java Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

We currently use the ColumnBuilder API in `HostColumnVector.java` to build batches on the spark side. The problem with this API is that as soon as an attempt is made to put host data onto the GPU, the host data is put in a try-with-resources block that closes it. If the creation of the GPU data fails, or a subsequent column in a batch fails to materialize, we cannot retry it (ColumnBuilder does not allow me to build an idempotent BatchBuilder).

This task is to likely add a BatchBuilder that allows us to control when/if the host data is freed, so we can work in retry semantics like https://github.com/NVIDIA/spark-rapids/issues/7851.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.