NVIDIA / NVIDIA/cudf

[BUG] Limit JSON reader input to 2GiB

Open
#17,017 3 comments 0 reactions 1 assignee Claimed by @shrshi View on GitHub
bug cuIO
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Due to limitations of certain algorithms such `cub::ExclusiveScan`, and json tree implementation, the input json can not exceed limit 2GiB. (INT_MAX)
The json_read_data_type INTEGRAL benchmark fail to run because of allocation out_of_memory, because the input json_size exceed 2GiB limit.
```
json_in.size() = 2167967896
total_temp_storage_bytes: 18446744073707336447
Fail: Unexpected error: std::bad_alloc: out_of_memory: RMM failure at:/home/coder/.conda/envs/rapids/include/rmm/mr/device/pool_memory_resource.hpp:277: Maximum pool size exceeded
```

**Steps/Code to reproduce bug**
`./benchmarks/JSON_READER_NVBENCH --profile --benchmark json_read_data_type --axis data_type=INTEGRAL`

**Expected behavior**
Either >2GiB should be batched multiple times with `read_json`, or error out that >2GiB sources are not supported.

**Additional context**
https://github.com/rapidsai/cudf/pull/16978

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.