dolthub / dolthub/doltpy

`write_pandas` silently discarding rows that contain NaNs

Open
#166 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
62
Forks
12
PR merge metrics
No merged PRs in 30d

Description

Hi there,

After a lot of debugging I just noticed that the `write_pandas` wrapper is silently discarding rows that contain NaN values.

[Here](https://github.com/dolthub/doltpy/blob/ecf23fd333aab37f136973d4f62ee983dbd41340/doltpy/cli/write.py#L46) is the point where this happens.

Is this behavior intended? Can't we toggle it providing this as additional parameter to the function?

Expected behavior:
In my case there are a lot of rows in my DataFrame that contains NaNs value but are just fine from the application logic perspective, and I would just like to store my entire DataFrame to `dolt` without dropping anything.

PS: even in case the drop is needed for any reason, it would make sense to print a warning to the user. In my case it was successfully concluding the write with no errors, but the fact that 50% of the content was silently dropped :)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.