IQSS / IQSS/dataverse

Feature request: create a harvested dataset via the API

Open
#7,330 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: API Feature: Harvesting FY27 Priority FY27 Roadmap GREI 3 Original size: 30 Type: Feature User Role: API User
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Some background: MIT is working on a research data registry using Dataverse as the repository, and as part of this project we want to ingest metadata into Dataverse such that it is indexed and searchable but the search result points to the source repository instead of a local Dataverse record. This is the exact functionality of harvested records, however we are building our own harvesting service because we need to perform actions not possible in the internal harvester, like parsing returned records to identify MIT-affiliated authors and enhancing the metadata with data from our local data warehouse.

In talking to the Dataverse tech team about this, the question rose of whether it would be feasible/useful to add the ability to create "harvested" datasets via the API. They suggested we add it here as an issue to prompt discussion...questions, comments, and suggestions welcome!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the existing internal harvester and dataset API behavior described in the issue, then clarify how externally created harvested records should be indexed and linked to their source repository. The issue names no files or tests; the work is done when the API design and metadata, search, and source-linking requirements are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.