ckan / ckan/ckanext-xloader

Bad encoding for resources uploaded into DataStore

Open
#174 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
58
Forks
67
Avg merge
23h 3m
Merged PRs (30d)
8

Description

### Version

CKAN 2.9.5; XLoader 0.11.0

### Describe the bug
Uploading a resource with accented characters are stored and are shown in the previews with bad encoding.

### Expected behavior
Uploading a resource with accented characters are stored and shown with the correct encoding.

### Additional details

Looks like somehow if the first rows of the file are accented the resource is being uploaded OK.

See the column 'Vwx' of the Attached two files after being xloaded to the datastore:

Good.csv : This is uploaded and show correctly.
Bad.csv: This is uploaded and shown incorrectly.

MessyTables behavior is set to False
> ckanext.xloader.just_load_with_messytables = False

[Good.csv](https://github.com/ckan/ckanext-xloader/files/9675182/Good.csv)
[Bad.csv](https://github.com/ckan/ckanext-xloader/files/9675183/Bad.csv)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the upload with Good.csv and Bad.csv while ckanext.xloader.just_load_with_messytables is False, then compare where their accented values diverge during XLoader processing or DataStore insertion. Done means accented characters from both files are stored and displayed correctly in the resource preview.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.