ckan / ckan/ideas

DataStore column name mapping

Open
#210 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
39
Forks
1
PR merge metrics
No merged PRs in 30d

Description

When importing a CSV into DataStore, the column names are set (by DataPusher/[Express Loader](https://github.com/davidread/ckanext-xloader)) to be the same in the database as they are in the CSV.

This is problematic:
* [In queries its confusing to have to double quote most column names](https://github.com/ckan/ckan/issues/2577)
* Postgres doesn't allow unicode characters. So DataPusher/Express Loader can't upload them.
* Postgres limit on column names length is 63 characters
* Repeated column names

How about we get DataPusher/Express Loader to make some sensible changes to column names before it goes into the database, and then the mapping is saved for:
* users can see it in the web interface
* users can get it via an API
* users can *edit* it in the web interface
* subsequent DataPusher/Express Loader runs reuse the same mapping if possible

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the DataStore import behavior and the DataPusher/Express Loader integration first. Define the scope for normalizing CSV column names and persisting their mapping. Done means the mapping handles PostgreSQL restrictions and duplicate names, is visible and editable in the web interface, available through an API, and reusable on later imports.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.