tursodatabase / tursodatabase/libsql

Treat ,, as NULL ,"", as empty string when importing CSV using sqlite shell

Open
#1,212 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
17.2k
Forks
531
Avg merge
1h 12m
Merged PRs (30d)
1

Description

I am trying to copy data from Postgres to libsql using the CSV format. SQLite already supports CSV imports and has good throughput, but has a major issue with CSV handling: SQLite does not respect the Postgres convention that a totally empty field ,, is NULL, and that a quoted empty field ,"", is '' EMPTY STRING - and does not seem to allow any configuration to set other sentinel values like '\N' or 'NULL' as null either.

Instead, because its the javascript of databases, it will insert an EMPTY STRING into an INTEGER column, and worse there’s no way to disambiguate nullable TEXT columns without referring to the original CSV file.

There’s a bunch of complaints on the SQLite forum about this, and no movement, although i think there are a few extensions. I see someone imported CSV import for the turso CLI, but it just delegates this to sqlite

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The linked turso-cli internal/cmd/group_flag.go delegates CSV import to SQLite; use that as context, then inspect the SQLite shell CSV import entry point. Compare the handling of unquoted ,, and quoted empty fields ,"", and determine how configurable sentinels such as \N or NULL should behave. Done means the intended NULL-versus-empty-string behavior is defined and covered by appropriate import tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, sqlite
Domain
cli, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.