IQSS / IQSS/dataverse

Feature Request: mySQL database ingest support and query API for DH projects and websites

Open
#11,420 3 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Feature
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Overview of the Feature Request
Originally posted in Google Group thread: "I published my findings on line as an integrated database/Wordpress-Commentpress book (https://stuehlingen.online/). Right now I am still able to maintain the website including its mySQL database. But I don't know for how long that will still be the case."
For example use cases, but many similar use cases from across community and some suggested setups and workflows see: https://groups.google.com/g/dataverse-community/c/Di2KXMgciUo/m/_pF79DFJCQAJ

What kind of user is the feature intended for?
(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)
API user, curator, depositor, Superuser , Admin

What inspired the request?
There isn't a lot of support built in for mySQL or Access databases , for example to read and ingest or preview these file formats in DV. A lot of DH research projects create or collect data in database format and are connected to project websites that support querying and retrieval. These projects and underlying databases are at risk of loss. Deposit in Dataverse is difficult for enabling reuse, without flattening the file into a spreadsheet for easy download and reuse. Some other suggestions mentioned in the thread include updated guidance and promotion of some workflows "e.g. Datasette ( https://datasette.io/ ) provides a querying interface to SQLite relational databases. One could conceivable upload an SQLite db file into Dataverse and then any user could take that file and load it locally into an instance of Datasette, or load the URL into Datasette Lite ( lite.datasette.io/ ). For example you can load this url: https://raw.githubusercontent.com/jpwhite3/northwind-SQLite3/main/dist/northwind.db
into Datasette Lite: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjpwhite3%2Fnorthwind-SQLite3%2Fmain%2Fdist%2Fnorthwind.db
This allows you to then query the database from the browser without much difficulty.
This only works with SQLite, so you'd have to convert mySQL to SQLite in this case."

Other ideas for connecting to queryable interfaces include:
https://shiny.posit.co/
and possible options mentioned in the following:
Database Preservation Toolkit (DBPTK): https://digital-strategy.ec.europa.eu/en/news/release-database-preservation-toolkit
Half-day session on database preservation at a hybrid workshop on May 14, 2025: https://digital-strategy.ec.europa.eu/en/events/preserving-digital-foundations-earchiving

What existing behavior do you want changed?
More support for ingest, identification, queryability, APIs, etc. for mySQL databases and building querable interfaces or HTML or R Shiny based websites or app connections

Any brand new behavior do you want to add to Dataverse?
Yes, ingest of this format similar to tabular data and geospatial data and more connected tools for querying databases in repository, perhaps using R

Any open or closed issues related to this feature request?
Not that I can tell

Are you thinking about creating a pull request for this feature?
Maybe, but not right now. Perhaps we can scope ideas with community members @philippconzett @janetm @adam3smith @jerabek and others

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no repository files, tests, or entry points. Start by scoping the requested MySQL ingest, identification, queryability, and API behavior against existing tabular and geospatial ingest, then identify the relevant extension points and tests. Done would require an agreed, testable feature scope rather than the broad set of workflows and integrations listed here.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, r, sqlite, wordpress
Domain
api, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.