ckan / ckan/ckanext-xloader

Non-intuitive messytables behavior

Open
#90 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
58
Forks
67
Avg merge
23h 3m
Merged PRs (30d)
8

Description

In testing XLoader and the Data Dictionary recently, I noticed that after setting just one column type to 'timestamp' on a new resource and re-submitting it, the whole Data Dictionary was populated with appropriate types. Checking the Datastore log, I realised that this was because our non-ISO dates (dd/mm/yyyy) had caused a direct PostgreSQL COPY to fail, so it fell back to using messytables - which meant that it guessed all of our types.

Although this behaviour is actually rather useful, it seems rather inconsistent that you get full type-guessing by *providing invalid input*, yet no such guessing if your data is entirely valid. It's also unfortunate that it presumably replaces any Data Dictionary overrides you may have entered manually.

So, there are two issues:

- If there are any existing Data Dictionary overrides, then messytables probably should not be permitted to overwrite them.

- How easily could the use of messytables be made configurable? It occurs to me that it would be useful to have at least a setting to use messytables on first creation of a resource, while attempting a fast COPY for further updates. I'm sure that there are also a lot of potential XLoader users (possibly even my own workplace) who would prefer the maximum-backwards-compatibility option of *always* using messytables, despite the more modest speed gains compared to DataPusher - and as a side benefit, this wouldn't result in the log warning when PostgreSQL COPY fails (which can confuse non-technical users).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing XLoader's handling of PostgreSQL COPY failures and its fallback to messytables, then examine how Data Dictionary overrides are applied during DataStore loads. The issue is not specific about files or tests; completion would need an agreed configuration model and behavior that preserves existing overrides.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.