blaze / blaze/odo

Importing CSV to Postgresql fails for versions < 9.0

Open
#486 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Postresql's COPY command syntax has changed from 8.x to 9.x. odo uses the new syntax, which produces a syntax error when running against 8.x and older. Below is a patch that probes the database version and applies the correct syntax. It also fixes a (suspected) bug in how encoding is inferred.

``` diff
--- odo/backends/sql_csv.py
+++ odo/backends/sql_csv.py
@@ -151,32 +151,55 @@
raise ValueError(
r'PostgreSQL does not support line terminators other than \n'
)
- return compiler.process(
- sa.text(
- """
- COPY {0} FROM :path (
- FORMAT CSV,
- DELIMITER :delimiter,
- NULL :na_value,
- QUOTE :quotechar,
- ESCAPE :escapechar,
- HEADER :header,
- ENCODING :encoding
- )
- """.format(compiler.preparer.format_table(element.element))
- ).bindparams(
- path=os.path.abspath(element.csv.path),
- delimiter=element.delimiter,
- na_value=element.na_value,
- quotechar=element.quotechar,
- escapechar=element.escapechar,
- header=element.header,
- encoding=element.encoding or element.bind(
- 'show client_encoding'
- ).execute().scalar()
- ),
- **kwargs
- )
+ postgres_version = element.bind.execute('select version()').scalar()
+ if int(postgres_version.split()[1].split('.')[0]) >= 9:
+ return compiler.process(
+ sa.text(
+ """
+ COPY {0} FROM :path (
+ FORMAT CSV,
+ DELIMITER :delimiter,
+ NULL :na_value,
+ QUOTE :quotechar,
+ ESCAPE :escapechar,
+ HEADER :header,
+ ENCODING :encoding
+ )
+ """.format(compiler.preparer.format_table(element.element))
+ ).bindparams(
+ path=os.path.abspath(element.csv.path),
+ delimiter=element.delimiter,
+ na_value=element.na_value,
+ quotechar=element.quotechar,
+ escapechar=element.escapechar,
+ header=element.header,
+ encoding=element.encoding or element.bind.execute(
+ 'show client_encoding'
+ ).scalar()
+ ),
+ **kwargs
+ )
+ else:
+ return compiler.process(
+ sa.text((
+ """
+ COPY {0} FROM :path
+ NULL :na_value
+ DELIMITER :delimiter
+ CSV %s
+ QUOTE :quotechar
+ ESCAPE :escapechar
+ """ % ('HEADER' if element.header else '')
+ ).format(compiler.preparer.format_table(element.element))
+ ).bindparams(
+ path=os.path.abspath(element.csv.path),
+ delimiter=element.delimiter,
+ na_value=element.na_value,
+ quotechar=element.quotechar,
+ escapechar=element.escapechar,
+ ),
+ **kwargs
+ )

try:
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the PostgreSQL CSV import code in odo/backends/sql_csv.py and reproduce the failure against a PostgreSQL version below 9.0. Verify the version-specific COPY syntax and the encoding lookup there; done means CSV imports work on both older and newer PostgreSQL versions without the reported syntax or encoding issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.