NVIDIA / NVIDIA/cudf

[BUG] Batched multi-source JSON reader does not require each batch to contain at least one JSON line

Open
#17,836 0 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
This bug is related to https://github.com/rapidsai/cudf/issues/17058
There are two scenarios in which the partial table constructed from a batch i.e., byte range across sources can result in an empty table:
1. When the last batch is an incomplete row: The following test fails with the error `Mismatch in JSON schema across batches in multi-source multi-batch JSON reader`
```
TEST_F(JsonReaderTest, Debug)
{
std::string data = R"(
{"a": "b"}
{"a": "b"}
{"a": "b"}
{"a": "b"}
)";
setenv("LIBCUDF_JSON_BATCH_SIZE", std::to_string(data.size() - 5).c_str(), 1);
auto opts =
cudf::io::json_reader_options::builder(cudf::io::source_info{data.data(), data.size()})
.lines(true)
.build();
auto res = cudf::io::read_json(opts);
unsetenv("LIBCUDF_JSON_BATCH_SIZE");
}
```
Since the penultimate batch reads the last row of the source, the byte range read in the last batch is empty and hence constructs an empty table.

2. When a batch contains a single JSON line of size > `INT_MAX`: In this case again, the partial table would be empty. However, the `INT_MAX` size constraint on the input buffer passed to the tokenizer precludes this error from appearing for now. A related corner case is documented in https://github.com/rapidsai/cudf/issues/17058, where the batch contains multiple lines, the last of which is incomplete and results in a buffer of size greater than 2GB chars.

**Expected behavior for the scenarios listed above**
1. Skip the empty table while concatenating partial tables from batches
2. Error out with a `CUDF_EXPECTS`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.