[BUG] No error detection in corrupted ORC files
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
This is related to https://github.com/rapidsai/cudf/issues/13460
After we write bad data, when we read it back in on the GPU we don't produce any errors. We don't crash. We just happily return corrupt data.
```
scala> spark.time(spark.read.parquet("./target/TMP_PAR").selectExpr("MIN(ts)", "MAX(ts)").show(false))
+--------------------------+--------------------------+
|min(ts) |max(ts) |
+--------------------------+--------------------------+
|2023-05-23 08:34:20.007655|2023-05-23 08:34:20.993544|
+--------------------------+--------------------------+
Time taken: 189 ms
scala> spark.time(spark.read.orc("./target/TMP_ORC").selectExpr("MIN(ts)", "MAX(ts)").show(false))
+-------------------+-------------------+
|min(ts) |max(ts) |
+-------------------+-------------------+
|2015-01-01 00:00:00|2015-01-01 00:00:08|
+-------------------+-------------------+
```
I realize that CUDF for performance reasons does not do much in the way of checks on the input data, but for input files we really should be doing something.
Contributor guide
Assessment
This issue has not been assessed yet.