blaze / blaze/blaze

Integer column has NA values in column when count values on HMDA large data

Open
#1,665 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.2k
Forks
389
PR merge metrics
No merged PRs in 30d

Description

hi, Sir

I am running the examples from [Anaconda blogs](https://www.anaconda.com/blog/developer-blog/introducing-blaze-hmda-practice/) with the large gz data it provides, my code is just minor changes as following
```
d = Data(r'd:\data\hmda_lar-2012.csv.gz')
columns = ['action_taken_name', 'agency_abbr', 'applicant_ethnicity_name',
'applicant_race_name_1', 'applicant_sex_name', 'county_name',
'loan_purpose_name', 'state_abbr']
d = d[columns]
d2 = d[d.state_abbr == 'NY']
dd=d2.action_taken_name.count_values()
print(dd.dshape)
dd.peek()
```

the code stops when dshape is printed, and error pop up as
```
Traceback (most recent call last):
File "C:\workspace\NeverCrash\src\demo.py", line 23, in
df=into(DataFrame,dd)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\interactive.py", line 404, in into
result = compute(b, return_type='native', **kwargs)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\interactive.py", line 195, in compute
return compute(expr, resources, **kwargs)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\compute\core.py", line 409, in compute
result = top_then_bottom_then_top_again_etc(expr3, d4, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\compute\core.py", line 153, in top_then_bottom_then_top_again_etc
return compute_down(expr, *leaf_data, **kwargs)
File "C:\IntelPython3\lib\site-packages\multipledispatch\dispatcher.py", line 164, in __call__
return func(*args, **kwargs)
File "C:\IntelPython3\lib\site-packages\blaze\compute\chunks.py", line 46, in compute_down
parts = list(map(curry(compute_chunk, chunk, chunk_expr), data))
File "C:\IntelPython3\lib\site-packages\odo\backends\csv.py", line 326, in _
for df in rest:
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1038, in __next__
return self.get_chunk()
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1102, in get_chunk
return self.read(nrows=size)
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1065, in read
ret = self._engine.read(nrows)
File "C:\IntelPython3\lib\site-packages\pandas\io\parsers.py", line 1828, in read
data = self._reader.read(nrows)
File "pandas\_libs\parsers.pyx", line 894, in pandas._libs.parsers.TextReader.read
File "pandas\_libs\parsers.pyx", line 928, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas\_libs\parsers.pyx", line 993, in pandas._libs.parsers.TextReader._read_rows
File "pandas\_libs\parsers.pyx", line 1122, in pandas._libs.parsers.TextReader._convert_column_data
File "pandas\_libs\parsers.pyx", line 1152, in pandas._libs.parsers.TextReader._convert_tokens
File "pandas\_libs\parsers.pyx", line 1219, in pandas._libs.parsers.TextReader._convert_with_dtype
ValueError: Integer column has NA values in column 67
```

Switching back the data (hmda-small.csv) in Blaze example , error disappeared.

My Pandas is 0.21.0 and Blaze is 0.10.1, could you help to look what is going wrong?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the reported example with the large hmda_lar-2012.csv.gz file and compare it with hmda-small.csv. Start by tracing the CSV chunk handling in odo/backends/csv.py and the Blaze compute path shown in the traceback; done means the large-data count_values operation completes without the integer-column NA failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.