Integer column has NA values in column when count values on HMDA large data
- Dominant language
- Python
- Stars
- 3.2k
- Forks
- 389
- PR merge metrics
- No merged PRs in 30d
Description
hi, Sir
I am running the examples from [Anaconda blogs](https://www.anaconda.com/blog/developer-blog/introducing-blaze-hmda-practice/) with the large gz data it provides, my code is just minor changes as following
```
d = Data(r'd:\data\hmda_lar-2012.csv.gz')
columns = ['action_taken_name', 'agency_abbr', 'applicant_ethnicity_name',
'applicant_race_name_1', 'applicant_sex_name', 'county_name',
'loan_purpose_name', 'state_abbr']
d = d[columns]
d2 = d[d.state_abbr == 'NY']
dd=d2.action_taken_name.count_values()
print(dd.dshape)
dd.peek()
```
the code stops when dshape is printed, and error pop up as
```
Traceback (most recent call last):
File "C:\workspace\NeverCrash\src\demo.py", line 23, in
df=into(DataFrame,dd)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\interactive.py", line 404, in into
result = compute(b, return_type='native', **kwargs)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\interactive.py", line 195, in compute
return compute(expr, resources, **kwargs)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\compute\core.py", line 409, in compute
result = top_then_bottom_then_top_again_etc(expr3, d4, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\compute\core.py", line 153, in top_then_bottom_then_top_again_etc
return compute_down(expr, *leaf_data, **kwargs)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\compute\chunks.py", line 46, in compute_down
parts = list(map(curry(compute_chunk, chunk, chunk_expr), data))
File "C:\IntelPython3\lib\site-packages\odo\backends\csv.py", line 326, in _
for df in rest:
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1038, in __next__
return self.get_chunk()
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1102, in get_chunk
return self.read(nrows=size)
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1065, in read
ret = self._engine.read(nrows)
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1828, in read
data = self._reader.read(nrows)
File "pandas\_libs\parsers.pyx", line 894, in pandas._libs.parsers.TextReader.read
File "pandas\_libs\parsers.pyx", line 928, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas\_libs\parsers.pyx", line 993, in pandas._libs.parsers.TextReader._read_rows
File "pandas\_libs\parsers.pyx", line 1122, in pandas._libs.parsers.TextReader._convert_column_data
File "pandas\_libs\parsers.pyx", line 1152, in pandas._libs.parsers.TextReader._convert_tokens
File "pandas\_libs\parsers.pyx", line 1219, in pandas._libs.parsers.TextReader._convert_with_dtype
ValueError: Integer column has NA values in column 67
```
Switching back the data (hmda-small.csv) in Blaze example , error disappeared.
My Pandas is 0.21.0 and Blaze is 0.10.1, could you help to look what is going wrong?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the reported example with the large hmda_lar-2012.csv.gz file and compare it with hmda-small.csv. Start by tracing the CSV chunk handling in odo/backends/csv.py and the Blaze compute path shown in the traceback; done means the large-data count_values operation completes without the integer-column NA failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100