blaze / blaze/odo

Support remote Postgres instances

Open
#457 0 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Unless I missing something, Odo currently relies on custom SQL via SQLAlchemy to execute a COPY statement that reads/writes from files local to the server. This may be disabled on some instances (eg AWS RDS), or not possible in a distributed environment where code can't run on the same instance as the source/target database.
https://github.com/blaze/odo/blob/master/odo/backends/sql_csv.py#L154
https://github.com/blaze/odo/blob/master/odo/backends/sql.py#L696

The Postgres COPY statement also supports reading/writing from stdin/stdout, which allows bulk reads/writes without requiring access to the local file system. The psycodb2 driver implements this through the cursor.copy_from/copy_to methods.

I'm not sure how to access this without going direct to the driver - eg not using SQLAlchemy - but doing so would allow Odo to be used in a much wider number of use cases (such as ours, where we have an ETL process bulk loading multiple different sources into a single target Postgres instance on AWS RDS.)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.