NVIDIA / NVIDIA/cudf

[BUG] JNI testORCWriteToBufferChunked fails if `device_write_async` just returns a future

Open
#9,430 8 comments 0 reactions 1 assignee Claimed by @vuule View on GitHub
bug Java
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

The JNI cuDF bindings have a custom writer sink. Recent changes to the `cudf::io::data_sink` api added a `device_write_async` method and we updated our custom sink to pass CI and retain the old behavior.

If I try to return a future in this API, and do the work we used to do inside of it, our java tests fail for Orc but not for Parquet.

```
diff --git a/java/src/main/native/src/TableJni.cpp b/java/src/main/native/src/TableJni.cpp
index 9e07f44..da036e9 100644
--- a/java/src/main/native/src/TableJni.cpp
+++ b/java/src/main/native/src/TableJni.cpp
@@ -145,9 +145,9 @@ public:

std::future device_write_async(void const *gpu_data, size_t size,
rmm::cuda_stream_view stream) override {
- // Call the sync version until figuring out how to write asynchronously.
- device_write(gpu_data, size, stream);
- return std::async(std::launch::deferred, [] {});
+ return std::async(std::launch::deferred, [=] {
+ device_write(gpu_data, size, stream);
+ });
}
```

Test failure:

```
ai.rapids.cudf.CudfException: cuDF failure at: /cudf/cpp/src/io/orc/reader_impl.cu:1321: Invalid index rowgroup stream data
at ai.rapids.cudf.Table.readORC(Native Method)
at ai.rapids.cudf.Table.readORC(Table.java:925)
at ai.rapids.cudf.TableTest.testORCWriteToBufferChunked(TableTest.java:7058)
```

Discussing with @devavret it seems that the issue is the value of `bytes_written`. The synchronous way, `bytes_written` in the sink was guaranteed to get updated in the same stack, but with the future approach, we have no such guarantee.

I am adding this issue to try and document why we worked around this, and if there's a fix to our custom sink or if it's an API that needs to get changed in cuDF.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.