NVIDIA / NVIDIA/cudf

[BUG] cuDF-Pandas reads missing values as <NA> whereas Pandas reads as NaN

Open
#18,504 1 comment 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
When using magic command to zero code accelerate Pandas, CSV are read differently.

In Pandas, missing values are read as NaN and in cuDF-Pandas, missing values are read as ``. This makes a big difference afterward if users are combining columns with `df[COL1].astype('str')+'_'+df[COL2].astype('str')`. In Pandas this creates a rich result that combines NaN and number. For example when `COL1=NaN` and `COL2=3` then the string combination is `nan_3.0` but in cuDF-Pandas the result is always `` and much information is lost in the combination.

**Steps/Code to reproduce bug**

csv_content = """a,b
1,4
2,5
3,6
,7
"""
with open("example.csv", "w") as f:
f.write(csv_content)
df = pd.read_csv("example.csv")

**Expected behavior**
It would be nice if both cuDF-Pandas matched the behavior of Pandas and read missing values as NaN

**Environment overview (please complete the following information)**
RAPIDS '25.02.02'

**Environment details**
Please run and paste the output of the `cudf/print_env.sh` script here, to gather any other relevant environment details

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.