wireservice / wireservice/csvkit
csvsql: Add option to use ON CONFLICT DO NOTHING/UPDATE
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.4k
- Forks
- 689
- Avg merge
- 1m
- Merged PRs (30d)
- 1
Description
I'm using csvsql to insert into to a Postregress database, and I'd like it to ignore any errors generated from unique constraints (basically skip importing the same data twice).
In Postgres and that would be handled like this:
INSERT INTO table (num1, num2) VALUES (1,1) ON CONFLICT DO UPDATE
However, it doesn't look like any of the current csvsql options allow me to modify the insert statement like this, since --prefix would add it direcly after INSERT and --after-insert executes a seperate statement. Postgres doesn't support the 'OR IGNORE' format that MySQL does.
Seems like a --suffix option to append something to the end of the query would be useful here, or is this a fundamental issue, since agate-sql doesn't support this?
Are there any alternatives for ignoring unique constraint insert attempt errors? Even just a flag to ignore any error from an individual insert statement and keep trying would help here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name files or tests. Start by tracing csvsql's INSERT generation and checking whether agate-sql exposes PostgreSQL conflict clauses; done should include a defined CLI behavior for DO NOTHING/UPDATE and coverage for duplicate-row imports.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- cli, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100