ckan / ckan/ckanext-xloader

Japanese kanjis in CSV header automatically replaced into Chinese Pinyins

Open
#145 10 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
58
Forks
67
Avg merge
23h 3m
Merged PRs (30d)
8

Description

When I uploaded a CSV file on a datastore via XLoader, I noticed that all Japaneses kanjis in the CSV header are replaced into Chinese Pinyins.

For example, when a XLoader loads such CSV file, we get this table.

```csv
id,緯度,経度
1,35.6824572,139.763119
2,35.6733858,139.7506726
```

| id | Wei Du | Jing Du |
| -- | -- | -- |
| 1 | 35.6824572 | 139.763119 |
| 2 | 35.6733858 | 139.7506726 |

*Note*: The pronunciation of `緯度` is `/ido/` and the pronunciation of `経度` is `/ke:do/` in Japanese. So you can notice these are quite far from the Chinese one.

But I want to get a table like this. So, those are should not be replaced.

| id | 緯度 | 経度 |
| -- | -- | -- |
| 1 | 35.6824572 | 139.763119 |
| 2 | 35.6733858 | 139.7506726 |

I know `unidecode` does that in [this function](https://github.com/ckan/ckanext-xloader/blob/master/ckanext/xloader/loader.py#L410-L418). But I don't have good idea not to encode Kanjis in a CSV header.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in ckanext/xloader/loader.py at lines 410-418, where the issue identifies unidecode being applied during CSV loading. Reproduce the example with Japanese headers and inspect the surrounding XLoader header-processing path. Done means Japanese kanji headers remain unchanged in the resulting DataStore table while the existing CSV loading behavior continues to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.