blaze / blaze/odo

potential mismatch of data type for csv column

Open
#461 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Due to the way of discovering the datashape of a csv file:

def discover_csv(c, nrows=1000, **kwargs):

by default, it only checks the first 1000 rows, so it is possible (For example, in a column with sparse data -- mostly 0 with scattered float number) that later rows may contain data of a different type (which does not appear in the 1000 sampled rows) and cause dtype error when actually loading the data.

More over, even when the user pass in a dshape parameter, it is not taken by the CSV constructor.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.