NULL (SQL) values for VARCHAR cols are not properly converted in pd.DataFrame
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
The above title is a big misleading since at the core of the issue is the translation of the dshape to the numpy dtype. However the title reflects (IMHO) a common use case for odo: (sql) Table <-> (pandas) DataFrame.
When odo "discovers" a nullable varchar column of a given length (n) it will create a datashape (datashape type): ?string[n]. For example a VARCHAR(10) NULL is ?string[10].
When odo "converts" the (sql) Table to the (pandas) DataFrame, it will first convert the Table to an Iterator, then chunksize it, move the chunk to numpy array, to a chunked DataFrame to then build the final DataFrame. The trip to numpy is the issue.
To create the numpy array with the correct dtype, odo translates the datashape into a valid numpy dtype. Here is our example:
`>>>odo.numpy_dtype.unit_to_dtype('?string[10]')`
`dtype('
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.