Japanese kanjis in CSV header automatically replaced into Chinese Pinyins
- Dominant language
- Python
- Stars
- 58
- Forks
- 67
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 8
Description
When I uploaded a CSV file on a datastore via XLoader, I noticed that all Japaneses kanjis in the CSV header are replaced into Chinese Pinyins.
For example, when a XLoader loads such CSV file, we get this table.
```csv
id,緯度,経度
1,35.6824572,139.763119
2,35.6733858,139.7506726
```
| id | Wei Du | Jing Du |
| -- | -- | -- |
| 1 | 35.6824572 | 139.763119 |
| 2 | 35.6733858 | 139.7506726 |
*Note*: The pronunciation of `緯度` is `/ido/` and the pronunciation of `経度` is `/ke:do/` in Japanese. So you can notice these are quite far from the Chinese one.
But I want to get a table like this. So, those are should not be replaced.
| id | 緯度 | 経度 |
| -- | -- | -- |
| 1 | 35.6824572 | 139.763119 |
| 2 | 35.6733858 | 139.7506726 |
I know `unidecode` does that in [this function](https://github.com/ckan/ckanext-xloader/blob/master/ckanext/xloader/loader.py#L410-L418). But I don't have good idea not to encode Kanjis in a CSV header.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in ckanext/xloader/loader.py at lines 410-418, where the issue identifies unidecode being applied during CSV loading. Reproduce the example with Japanese headers and inspect the surrounding XLoader header-processing path. Done means Japanese kanji headers remain unchanged in the resulting DataStore table while the existing CSV loading behavior continues to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100