NVIDIA / NVIDIA/cudf

[BUG] No error detection in corrupted ORC files

Open
#13,461 2 comments 0 reactions 1 assignee Claimed by @vuule View on GitHub
0 - Backlog bug cuIO libcudf Spark
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
This is related to https://github.com/rapidsai/cudf/issues/13460

After we write bad data, when we read it back in on the GPU we don't produce any errors. We don't crash. We just happily return corrupt data.

```
scala> spark.time(spark.read.parquet("./target/TMP_PAR").selectExpr("MIN(ts)", "MAX(ts)").show(false))
+--------------------------+--------------------------+
|min(ts) |max(ts) |
+--------------------------+--------------------------+
|2023-05-23 08:34:20.007655|2023-05-23 08:34:20.993544|
+--------------------------+--------------------------+

Time taken: 189 ms

scala> spark.time(spark.read.orc("./target/TMP_ORC").selectExpr("MIN(ts)", "MAX(ts)").show(false))
+-------------------+-------------------+
|min(ts) |max(ts) |
+-------------------+-------------------+
|2015-01-01 00:00:00|2015-01-01 00:00:08|
+-------------------+-------------------+
```

I realize that CUDF for performance reasons does not do much in the way of checks on the input data, but for input files we really should be doing something.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.