MagicStack / MagicStack/asyncpg
Support for `WHERE` clause when usng `copy_to_table` method
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 468
- PR merge metrics
- No merged PRs in 30d
Description
Hi there!
Since version 12, PostgreSQL has supported a `WHERE` clause within `COPY ... FROM` statements:
```
COPY table_name [ ( column_name [, ...] ) ]
FROM { 'filename' | PROGRAM 'command' | STDIN }
[ [ WITH ] ( option [, ...] ) ]
[ WHERE condition ]
```
It functions similarly to the `WHERE` clause used within `SELECT` statements, subject to some restrictions regarding subqueries.
Recently I've found myself in a situation where I am performing a bulk insert of CSV-formatted data from a remote machine, but would like to skip certain rows - such as those containing null values. From what I can tell, the existing `Connection.copy_to_table` helper doesn't support this, and I'd need to drop down to private API (the underlying `Connection._protocol._copy_in` machinery)
I'd appreciate some publicly exposed way of doing this - the simplest approach that comes to mind would be adding a `where` kwarg to `Connection.copy_to_table` as well as the `Connection._format_copy_opts` method. Usage could then look something along the lines of
```py
await conn.copy_to_table("cool_table", source="important_file.csv", where="important_value = 42")
```
I'm not sure if this is satisfactory, though. There would also need to be some consideration regarding earlier Postgres versions that don't support `COPY ... FROM` using a `WHERE` clause.
Please let me know your thoughts - I'm happy to open a PR for implementation if necessary.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting Connection.copy_to_table and Connection._format_copy_opts, then trace how they reach the private Connection._protocol._copy_in machinery. Check how PostgreSQL version compatibility is handled and identify the relevant tests before deciding how the public option should behave. Done means a supported way to apply a COPY ... FROM WHERE condition, with appropriate behavior for earlier PostgreSQL versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100