Add or document streamlined workflow for importing Datasette csv / json exports
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 172
- Avg merge
- 9m
- Merged PRs (30d)
- 1
Description
I'm working on some small front-end enhancements to the laion-aesthetic-datasette project, and I wanted to partially populate a database directly using exports from the existing Datasette instance instead of downloading the parquet files and creating my own multi-GB database.
There have been a number of small issues that are certainly related to my relative lack of familiarity with the toolkit, but that are still surprising.
For example: a CSV export of the images table (http://laion-aesthetic.datasette.io/laion-aesthetic-6pls.csv?sql=select+rowid%2C+url%2C+text%2C+domain_id%2C+width%2C+height%2C+similarity%2C+punsafe%2C+pwatermark%2C+aesthetic%2C+hash%2C+__index_level_0__+from+images+order+by+random%28%29+limit+100) has nested single quotes, double quotes, and commas that aren't handled by rows_from_file. Similarly, the json output has to be manually transformed to add the column names and remove extraneous information before sqlite_utils can import it.
I was able to work through these issues, but as an enhancement it would be really helpful to create or document a clear workflow that avoids the friction of this data transformation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the CSV and JSON export examples in the issue and inspect how rows_from_file handles them. Compare the exported shapes with what sqlite-utils import expects. Done would be either a supported streamlined import workflow or documentation that clearly explains the required transformation steps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, sqlite
- Domain
- cli, database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100