NVIDIA / NVIDIA/cudf

[BUG] Performance difference between cudf and dask_cudf when reading jsonl files

Open
#10,867 3 comments 1 reaction 0 assignees View on GitHub
0 - Backlog bug dask Performance Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Hi, I have noticed a difference in performance when reading a jsonl file with cudf and dask_cudf.

In both cases, I will be using only 1 GPU.

I have the following files (see details below):
- jsonl_cudf.py
- jsonl_dask_cudf.py

Please find below the execution time when I run them on a DGX1 v100 (16GBs):
```
(rapids) root@6ccf9a94ad0e:/rapids/notebooks/host# python jsonl_cudf.py
4.183666706085205
(rapids) root@6ccf9a94ad0e:/rapids/notebooks/host# python jsonl_dask_cudf.py
6.8754589557647705
```

The scripts content is as follows:
`json_cudf.py`

```
import cudf
import time

start = time.time()
df = cudf.read_json("x00_002GB.jsonl", lines=True)
end = time.time()
print(end - start)
```

and
`jsonl_dask_cudf.py`

```
import dask_cudf
import time

start = time.time()
df = dask_cudf.read_json("x00_002GB.jsonl", lines=True)
end = time.time()
print(end - start)
```

**Steps/Code to reproduce bug**
Hi @shwina , as discussed in the Slack channel, I will send you an email with the link to the dataset used. Thanks!

**Expected behavior**
Not such a huge difference in performance.

**Environment overview (please complete the following information)**
DGX-A100, cuda 11.5, rapids 22.04

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.