Feature Request: mySQL database ingest support and query API for DH projects and websites
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Overview of the Feature Request
Originally posted in Google Group thread: "I published my findings on line as an integrated database/Wordpress-Commentpress book (https://stuehlingen.online/). Right now I am still able to maintain the website including its mySQL database. But I don't know for how long that will still be the case."
For example use cases, but many similar use cases from across community and some suggested setups and workflows see: https://groups.google.com/g/dataverse-community/c/Di2KXMgciUo/m/_pF79DFJCQAJ
What kind of user is the feature intended for?
(Example users roles: API User, Curator, Depositor, Guest, Superuser, Sysadmin)
API user, curator, depositor, Superuser , Admin
What inspired the request?
There isn't a lot of support built in for mySQL or Access databases , for example to read and ingest or preview these file formats in DV. A lot of DH research projects create or collect data in database format and are connected to project websites that support querying and retrieval. These projects and underlying databases are at risk of loss. Deposit in Dataverse is difficult for enabling reuse, without flattening the file into a spreadsheet for easy download and reuse. Some other suggestions mentioned in the thread include updated guidance and promotion of some workflows "e.g. Datasette ( https://datasette.io/ ) provides a querying interface to SQLite relational databases. One could conceivable upload an SQLite db file into Dataverse and then any user could take that file and load it locally into an instance of Datasette, or load the URL into Datasette Lite ( lite.datasette.io/ ). For example you can load this url: https://raw.githubusercontent.com/jpwhite3/northwind-SQLite3/main/dist/northwind.db
into Datasette Lite: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Fjpwhite3%2Fnorthwind-SQLite3%2Fmain%2Fdist%2Fnorthwind.db
This allows you to then query the database from the browser without much difficulty.
This only works with SQLite, so you'd have to convert mySQL to SQLite in this case."
Other ideas for connecting to queryable interfaces include:
https://shiny.posit.co/
and possible options mentioned in the following:
Database Preservation Toolkit (DBPTK): https://digital-strategy.ec.europa.eu/en/news/release-database-preservation-toolkit
Half-day session on database preservation at a hybrid workshop on May 14, 2025: https://digital-strategy.ec.europa.eu/en/events/preserving-digital-foundations-earchiving
What existing behavior do you want changed?
More support for ingest, identification, queryability, APIs, etc. for mySQL databases and building querable interfaces or HTML or R Shiny based websites or app connections
Any brand new behavior do you want to add to Dataverse?
Yes, ingest of this format similar to tabular data and geospatial data and more connected tools for querying databases in repository, perhaps using R
Any open or closed issues related to this feature request?
Not that I can tell
Are you thinking about creating a pull request for this feature?
Maybe, but not right now. Perhaps we can scope ideas with community members @philippconzett @janetm @adam3smith @jerabek and others
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no repository files, tests, or entry points. Start by scoping the requested MySQL ingest, identification, queryability, and API behavior against existing tabular and geospatial ingest, then identify the relevant extension points and tests. Done would require an agreed, testable feature scope rather than the broad set of workflows and integrations listed here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, r, sqlite, wordpress
- Domain
- api, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100