blaze / blaze/odo

NULL (SQL) values for VARCHAR cols are not properly converted in pd.DataFrame

Open
#536 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

The above title is a big misleading since at the core of the issue is the translation of the dshape to the numpy dtype. However the title reflects (IMHO) a common use case for odo: (sql) Table <-> (pandas) DataFrame.
When odo "discovers" a nullable varchar column of a given length (n) it will create a datashape (datashape type): ?string[n]. For example a VARCHAR(10) NULL is ?string[10].
When odo "converts" the (sql) Table to the (pandas) DataFrame, it will first convert the Table to an Iterator, then chunksize it, move the chunk to numpy array, to a chunked DataFrame to then build the final DataFrame. The trip to numpy is the issue.

To create the numpy array with the correct dtype, odo translates the datashape into a valid numpy dtype. Here is our example:
`>>>odo.numpy_dtype.unit_to_dtype('?string[10]')`
`dtype('

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.