wireservice / wireservice/csvkit

csvsql: Add option to use ON CONFLICT DO NOTHING/UPDATE

Open
#1,186 2 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

csvsql feature: upstream
Dominant language
Python
Stars
6.4k
Forks
689
Avg merge
1m
Merged PRs (30d)
1

Description

I'm using csvsql to insert into to a Postregress database, and I'd like it to ignore any errors generated from unique constraints (basically skip importing the same data twice).

In Postgres and that would be handled like this:

INSERT INTO table (num1, num2) VALUES (1,1) ON CONFLICT DO UPDATE

However, it doesn't look like any of the current csvsql options allow me to modify the insert statement like this, since --prefix would add it direcly after INSERT and --after-insert executes a seperate statement. Postgres doesn't support the 'OR IGNORE' format that MySQL does.

Seems like a --suffix option to append something to the end of the query would be useful here, or is this a fundamental issue, since agate-sql doesn't support this?

Are there any alternatives for ignoring unique constraint insert attempt errors? Even just a flag to ignore any error from an individual insert statement and keep trying would help here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not name files or tests. Start by tracing csvsql's INSERT generation and checking whether agate-sql exposes PostgreSQL conflict clauses; done should include a defined CLI behavior for DO NOTHING/UPDATE and coverage for duplicate-row imports.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
cli, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.