Non-intuitive messytables behavior
- Dominant language
- Python
- Stars
- 58
- Forks
- 67
- Avg merge
- 23h 3m
- Merged PRs (30d)
- 8
Description
In testing XLoader and the Data Dictionary recently, I noticed that after setting just one column type to 'timestamp' on a new resource and re-submitting it, the whole Data Dictionary was populated with appropriate types. Checking the Datastore log, I realised that this was because our non-ISO dates (dd/mm/yyyy) had caused a direct PostgreSQL COPY to fail, so it fell back to using messytables - which meant that it guessed all of our types.
Although this behaviour is actually rather useful, it seems rather inconsistent that you get full type-guessing by *providing invalid input*, yet no such guessing if your data is entirely valid. It's also unfortunate that it presumably replaces any Data Dictionary overrides you may have entered manually.
So, there are two issues:
- If there are any existing Data Dictionary overrides, then messytables probably should not be permitted to overwrite them.
- How easily could the use of messytables be made configurable? It occurs to me that it would be useful to have at least a setting to use messytables on first creation of a resource, while attempting a fast COPY for further updates. I'm sure that there are also a lot of potential XLoader users (possibly even my own workplace) who would prefer the maximum-backwards-compatibility option of *always* using messytables, despite the more modest speed gains compared to DataPusher - and as a side benefit, this wouldn't result in the log warning when PostgreSQL COPY fails (which can confuse non-technical users).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing XLoader's handling of PostgreSQL COPY failures and its fallback to messytables, then examine how Data Dictionary overrides are applied during DataStore loads. The issue is not specific about files or tests; completion would need an agreed configuration model and behavior that preserves existing overrides.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100