IQSS / IQSS/dataverse

Inconsistent data between the harvesting server and client

Open
#12,406 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: Harvesting Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

After harvesting datasets we are finding differences between client and server. This mainly consider dates.

When I check api by calling /api/datasets/export?exporter=dataverse_json&persistentId=doi%3A10.57903/UJ/RH0J5L

I can see proper data in fields like lastUpdateTime and releaseTime.

However, the Dataverse UI shows the dates related to the harvesting moment.

Those wrong dates I can find in db by SELECT * FROM dvobject WHERE dtype = 'Dataset' AND authority = '10.xxx' AND identifier = 'xxx'

What is more sometimes even the title is wrong but with the same pattern: dataverse_json show correct values, but wrong one are stored in db and displayed to user.

This is confusing and should not happen. Problem is not related to dataset versions.

  • When does this issue occur?

This is a bug related to harvesting and was noticed in the context of dates displayed to Harvester users.

When trying to fix by deleting harvesting client/server and reharvesting data from Dataverse collection - all datasets received one date on harvester: current date (as mentioned above: misleading dates are in dvobject as lastUpdateTime and releaseTime - when I correct them manually in db, then they are correctly displayed for end user)

  • Which page(s) does it occurs on?
    Harvester pages visible for users.

  • What happens?

Harvested data are incorrect.

  • To whom does it occur (all users, curators, superusers)?

all

  • What did you expect to happen?

harvesting client and server should display same data for users.

Which version of Dataverse are you using?
6.7.1 at this moment.

Screenshots:

dataverse_json:

Image Image

displayed on harvester page:
Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the mismatch through /api/datasets/export?exporter=dataverse_json and the harvester pages, then compare those values with the corresponding dvobject row. Focus on the harvesting flow and the lastUpdateTime, releaseTime, and title fields, including a delete-and-reharvest cycle. Done means harvested client and server data match and the UI shows the source dates and title.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.