[BUG] Limit JSON reader input to 2GiB
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Due to limitations of certain algorithms such `cub::ExclusiveScan`, and json tree implementation, the input json can not exceed limit 2GiB. (INT_MAX)
The json_read_data_type INTEGRAL benchmark fail to run because of allocation out_of_memory, because the input json_size exceed 2GiB limit.
```
json_in.size() = 2167967896
total_temp_storage_bytes: 18446744073707336447
Fail: Unexpected error: std::bad_alloc: out_of_memory: RMM failure at:/home/coder/.conda/envs/rapids/include/rmm/mr/device/pool_memory_resource.hpp:277: Maximum pool size exceeded
```
**Steps/Code to reproduce bug**
`./benchmarks/JSON_READER_NVBENCH --profile --benchmark json_read_data_type --axis data_type=INTEGRAL`
**Expected behavior**
Either >2GiB should be batched multiple times with `read_json`, or error out that >2GiB sources are not supported.
**Additional context**
https://github.com/rapidsai/cudf/pull/16978
Contributor guide
Assessment
This issue has not been assessed yet.