Fails to process XLSX with old dates
- Dominant language
- Python
- Stars
- 82
- Forks
- 157
- PR merge metrics
- No merged PRs in 30d
Description
This looks more like an issue with messytables, but because the error message is so obscure, there seems like improvements to be made in how datapusher handles errors with guessing the header type.
When uploading the [IOLP Buildings dataset](https://inventory.data.gov/dataset/9a9e946e-124e-46f5-a934-e458a6c1c2b2/resource/e7922d4d-aed1-44b2-85c4-5cffcb9bfe94/download/iolp-buildings.xlsx), CKAN shows this error:
> Error: Process completed but unable to post to result_url
I think the underlying issue is that there are dates from before 1970. This dataset contains a date: 12/31/1939.
In the datapusher logs:
```
[Thu Jan 02 19:59:05.903277 2020] [:error] [pid 3138:tid 139758831396608] Fetching from: https://inventory.data.gov/dataset/9a9e946e-124e-46f5-a934-e458a6c1c2b2/resource/e7922d4d-aed1-44b2-85c4-5cffcb9bfe94/download/iolp-buildings.xlsx
[Thu Jan 02 19:59:08.762159 2020] [:error] [pid 3138:tid 139758831396608] Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
[Thu Jan 02 19:59:08.762197 2020] [:error] [pid 3138:tid 139758831396608] Traceback (most recent call last):
[Thu Jan 02 19:59:08.762202 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
[Thu Jan 02 19:59:08.762207 2020] [:error] [pid 3138:tid 139758831396608] retval = job.func(*job.args, **job.kwargs)
[Thu Jan 02 19:59:08.762212 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/src/datapusher/datapusher/jobs.py", line 380, in push_to_datastore
[Thu Jan 02 19:59:08.762216 2020] [:error] [pid 3138:tid 139758831396608] offset, headers = messytables.headers_guess(row_set.sample)
[Thu Jan 02 19:59:08.762221 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/lib/python2.7/site-packages/messytables/headers.py", line 28, in headers_guess
[Thu Jan 02 19:59:08.762225 2020] [:error] [pid 3138:tid 139758831396608] rows = list(rows)
[Thu Jan 02 19:59:08.762229 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/lib/python2.7/site-packages/messytables/core.py", line 236, in __iter__
[Thu Jan 02 19:59:08.762234 2020] [:error] [pid 3138:tid 139758831396608] for row in self.raw(sample=sample):
[Thu Jan 02 19:59:08.762238 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/lib/python2.7/site-packages/messytables/excel.py", line 112, in raw
[Thu Jan 02 19:59:08.762242 2020] [:error] [pid 3138:tid 139758831396608] row.append(XLSCell.from_xlrdcell(cell, self.sheet, colnum, rownum))
[Thu Jan 02 19:59:08.762247 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/lib/python2.7/site-packages/messytables/excel.py", line 127, in from_xlrdcell
[Thu Jan 02 19:59:08.762259 2020] [:error] [pid 3138:tid 139758831396608] xlrd.xldate_as_tuple(value, sheet.book.datemode)
[Thu Jan 02 19:59:08.762263 2020] [:error] [pid 3138:tid 139758831396608] File "/usr/lib/datapusher/lib/python2.7/site-packages/xlrd/xldate.py", line 96, in xldate_as_tuple
[Thu Jan 02 19:59:08.762267 2020] [:error] [pid 3138:tid 139758831396608] raise XLDateNegative(xldate)
[Thu Jan 02 19:59:08.762271 2020] [:error] [pid 3138:tid 139758831396608] XLDateNegative: -6938.0
```
## How to reproduce
1. Download https://inventory.data.gov/dataset/9a9e946e-124e-46f5-a934-e458a6c1c2b2/resource/e7922d4d-aed1-44b2-85c4-5cffcb9bfe94/download/iolp-buildings.xlsx
1. Create a new dataset in CKAN
1. Upload to Datastore
## Expected behavior
Datapusher falls back to treating the problematic columns as simple text OR job fails with a clear message indicating what the issue is.
## Actual
> Error: Process completed but unable to post to result_url
Unclear if data was successfully pushed or not.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.