Importing CSV to Postgresql fails for versions < 9.0
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
Postresql's COPY command syntax has changed from 8.x to 9.x. odo uses the new syntax, which produces a syntax error when running against 8.x and older. Below is a patch that probes the database version and applies the correct syntax. It also fixes a (suspected) bug in how encoding is inferred.
``` diff
--- odo/backends/sql_csv.py
+++ odo/backends/sql_csv.py
@@ -151,32 +151,55 @@
raise ValueError(
r'PostgreSQL does not support line terminators other than \n'
)
- return compiler.process(
- sa.text(
- """
- COPY {0} FROM :path (
- FORMAT CSV,
- DELIMITER :delimiter,
- NULL :na_value,
- QUOTE :quotechar,
- ESCAPE :escapechar,
- HEADER :header,
- ENCODING :encoding
- )
- """.format(compiler.preparer.format_table(element.element))
- ).bindparams(
- path=os.path.abspath(element.csv.path),
- delimiter=element.delimiter,
- na_value=element.na_value,
- quotechar=element.quotechar,
- escapechar=element.escapechar,
- header=element.header,
- encoding=element.encoding or element.bind(
- 'show client_encoding'
- ).execute().scalar()
- ),
- **kwargs
- )
+ postgres_version = element.bind.execute('select version()').scalar()
+ if int(postgres_version.split()[1].split('.')[0]) >= 9:
+ return compiler.process(
+ sa.text(
+ """
+ COPY {0} FROM :path (
+ FORMAT CSV,
+ DELIMITER :delimiter,
+ NULL :na_value,
+ QUOTE :quotechar,
+ ESCAPE :escapechar,
+ HEADER :header,
+ ENCODING :encoding
+ )
+ """.format(compiler.preparer.format_table(element.element))
+ ).bindparams(
+ path=os.path.abspath(element.csv.path),
+ delimiter=element.delimiter,
+ na_value=element.na_value,
+ quotechar=element.quotechar,
+ escapechar=element.escapechar,
+ header=element.header,
+ encoding=element.encoding or element.bind.execute(
+ 'show client_encoding'
+ ).scalar()
+ ),
+ **kwargs
+ )
+ else:
+ return compiler.process(
+ sa.text((
+ """
+ COPY {0} FROM :path
+ NULL :na_value
+ DELIMITER :delimiter
+ CSV %s
+ QUOTE :quotechar
+ ESCAPE :escapechar
+ """ % ('HEADER' if element.header else '')
+ ).format(compiler.preparer.format_table(element.element))
+ ).bindparams(
+ path=os.path.abspath(element.csv.path),
+ delimiter=element.delimiter,
+ na_value=element.na_value,
+ quotechar=element.quotechar,
+ escapechar=element.escapechar,
+ ),
+ **kwargs
+ )
try:
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the PostgreSQL CSV import code in odo/backends/sql_csv.py and reproduce the failure against a PostgreSQL version below 9.0. Verify the version-specific COPY syntax and the encoding lookup there; done means CSV imports work on both older and newer PostgreSQL versions without the reported syntax or encoding issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100