[BUG] JNI testORCWriteToBufferChunked fails if `device_write_async` just returns a future
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
The JNI cuDF bindings have a custom writer sink. Recent changes to the `cudf::io::data_sink` api added a `device_write_async` method and we updated our custom sink to pass CI and retain the old behavior.
If I try to return a future in this API, and do the work we used to do inside of it, our java tests fail for Orc but not for Parquet.
```
diff --git a/java/src/main/native/src/TableJni.cpp b/java/src/main/native/src/TableJni.cpp
index 9e07f44..da036e9 100644
--- a/java/src/main/native/src/TableJni.cpp
+++ b/java/src/main/native/src/TableJni.cpp
@@ -145,9 +145,9 @@ public:
std::future device_write_async(void const *gpu_data, size_t size,
rmm::cuda_stream_view stream) override {
- // Call the sync version until figuring out how to write asynchronously.
- device_write(gpu_data, size, stream);
- return std::async(std::launch::deferred, [] {});
+ return std::async(std::launch::deferred, [=] {
+ device_write(gpu_data, size, stream);
+ });
}
```
Test failure:
```
ai.rapids.cudf.CudfException: cuDF failure at: /cudf/cpp/src/io/orc/reader_impl.cu:1321: Invalid index rowgroup stream data
at ai.rapids.cudf.Table.readORC(Native Method)
at ai.rapids.cudf.Table.readORC(Table.java:925)
at ai.rapids.cudf.TableTest.testORCWriteToBufferChunked(TableTest.java:7058)
```
Discussing with @devavret it seems that the issue is the value of `bytes_written`. The synchronous way, `bytes_written` in the sink was guaranteed to get updated in the same stack, but with the future approach, we have no such guarantee.
I am adding this issue to try and document why we worked around this, and if there's a fix to our custom sink or if it's an API that needs to get changed in cuDF.
Contributor guide
Assessment
This issue has not been assessed yet.