tursodatabase / tursodatabase/libsql
Treat ,, as NULL ,"", as empty string when importing CSV using sqlite shell
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 17.2k
- Forks
- 531
- Avg merge
- 1h 12m
- Merged PRs (30d)
- 1
Description
I am trying to copy data from Postgres to libsql using the CSV format. SQLite already supports CSV imports and has good throughput, but has a major issue with CSV handling: SQLite does not respect the Postgres convention that a totally empty field ,, is NULL, and that a quoted empty field ,"", is '' EMPTY STRING - and does not seem to allow any configuration to set other sentinel values like '\N' or 'NULL' as null either.
Instead, because its the javascript of databases, it will insert an EMPTY STRING into an INTEGER column, and worse there’s no way to disambiguate nullable TEXT columns without referring to the original CSV file.
There’s a bunch of complaints on the SQLite forum about this, and no movement, although i think there are a few extensions. I see someone imported CSV import for the turso CLI, but it just delegates this to sqlite
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The linked turso-cli internal/cmd/group_flag.go delegates CSV import to SQLite; use that as context, then inspect the SQLite shell CSV import entry point. Compare the handling of unquoted ,, and quoted empty fields ,"", and determine how configurable sentinels such as \N or NULL should behave. Done means the intended NULL-versus-empty-string behavior is defined and covered by appropriate import tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, sqlite
- Domain
- cli, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100