ckan / ckan/datapusher

I met bug when my csv header began with comma

Open
#138 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
82
Forks
157
PR merge metrics
No merged PRs in 30d

Description

### situation
When I upload a file like this,the first header is '' since the line begins with comma.(See the up left empty cell)
![image](https://user-images.githubusercontent.com/5809588/28064352-d8d33e8c-6666-11e7-8012-69b301c61a8d.png)

Then DataPusher raise error as `The data was invalid (for example: a numeric value is out of range or was inserted into a text field). `
### debug
The directly reason of the error is `headers_dicts` in DataPusher:
I log the headers_dicts then it shows:
```
[
{
'type': u'numeric',
'id': u'name'
},
{
'type': u'text',
'id': u'total'
},
{
'type': u'numeric',
'id': u'male_share'
},
{
'type': u'numeric',
'id': u'female_share'
},
{
'type': u'numeric',
'id': u'gap'
}
]
```
Obviously there is some shifting thing cuz the `name` should be `text` and the `total` should be `numeric`

I read the code :
1. `offset, headers = messytables.headers_guess(row_set.sample)`
2. `types = messytables.type_guess(row_set.sample, types=TYPES, strict=False)`
The first empty header would be stored as u'',It's totally fine.
Here len(types)==len(headers)
3. `headers = [header.strip() for header in headers if header.strip()]`
Then this line dropped the u'',but didn't drop correspond type, that is ,len(types)== len(headers) -1

4. `headers_dicts = [dict(id=field[0], type=TYPE_MAPPING[str(field[1])]) for field in zip(headers, types)]`
Finally in this line we get a wrong headers_dicts, `zip`func would not raise error even if parameters have different length.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.