MagicStack / MagicStack/asyncpg

Support for `WHERE` clause when usng `copy_to_table` method

Open
#939 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.1k
Forks
468
PR merge metrics
No merged PRs in 30d

Description

Hi there!

Since version 12, PostgreSQL has supported a `WHERE` clause within `COPY ... FROM` statements:
```
COPY table_name [ ( column_name [, ...] ) ]
FROM { 'filename' | PROGRAM 'command' | STDIN }
[ [ WITH ] ( option [, ...] ) ]
[ WHERE condition ]
```
It functions similarly to the `WHERE` clause used within `SELECT` statements, subject to some restrictions regarding subqueries.

Recently I've found myself in a situation where I am performing a bulk insert of CSV-formatted data from a remote machine, but would like to skip certain rows - such as those containing null values. From what I can tell, the existing `Connection.copy_to_table` helper doesn't support this, and I'd need to drop down to private API (the underlying `Connection._protocol._copy_in` machinery)

I'd appreciate some publicly exposed way of doing this - the simplest approach that comes to mind would be adding a `where` kwarg to `Connection.copy_to_table` as well as the `Connection._format_copy_opts` method. Usage could then look something along the lines of
```py
await conn.copy_to_table("cool_table", source="important_file.csv", where="important_value = 42")
```
I'm not sure if this is satisfactory, though. There would also need to be some consideration regarding earlier Postgres versions that don't support `COPY ... FROM` using a `WHERE` clause.

Please let me know your thoughts - I'm happy to open a PR for implementation if necessary.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting Connection.copy_to_table and Connection._format_copy_opts, then trace how they reach the private Connection._protocol._copy_in machinery. Check how PostgreSQL version compatibility is handled and identify the relevant tests before deciding how the public option should behave. Done means a supported way to apply a COPY ... FROM WHERE condition, with appropriate behavior for earlier PostgreSQL versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.