Problems with Harvested Records

Hello from NIE,

We have an arrangement where NTU Dataverse (https://researchdata.ntu.edu.sg/) harvests records from NIE Dataverse (https://researchdata.nie.edu.sg/).

NIE is using Dataverse v5.8. It was upgraded from v5.4 in June 2022.

In November 2022, NTU reported that their records harvested from NIE do not contain some information, for eg. "versionnumber" was null even though "versionstate" was "RELEASED" in the table "datasetversion".

However, when NIE checked the same set of records in NIE Dataverse instance via SQL, the "versionnumber" was not null when "versionstate" is "RELEASED" in the table "datasetversion".

Apparently, the harvesting via OAI did not capture the "versionnumber" correctly.

• To check if the above issue is specific to NIE Dataverse instance, we used the NIE Dataverse Test Server (https://researchdatatest.nie.edu.sg/) to harvest records from NTU Dataverse. In our Test Server, the records harvested from NTU Dataverse have the same problem: "versionnumber" was null even though "versionstate" is "RELEASED" in the table "datasetversion".

Below is the client config (as at 5 Jan 2023) of NTU Dataverse that harvests records from NIE Dataverse. Many thanks!

{
  "status": "OK",
  "data": {
    "harvestingClients": [
      {
        "nickName": "harvested-nie",
        "dataverseAlias": "harvested-nie",
        "type": "oai",
        "harvestUrl": "https://researchdata.nie.edu.sg/oai",
        "archiveUrl": "https://researchdata.nie.edu.sg",
        "archiveDescription": "This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data.",
        "metadataFormat": "oai_dc",
        "set": "N/A",
        "schedule": "none",
        "status": "inActive",
        "lastHarvest": "Sun Nov 13 00:00:00 SGT 2022",
        "lastResult": "SUCCESS",
        "lastSuccessful": "Sun Nov 13 00:00:00 SGT 2022",
        "lastNonEmpty": "Sun Oct 02 00:00:00 SGT 2022",
        "lastDatasetsHarvested": "2",
        "lastDatasetsDeleted": "0",
        "lastDatasetsFailed": "0"
      },
      {
        "nickName": "NIE_dataverse_ddi",
        "dataverseAlias": "harvested-nie",
        "type": "oai",
        "harvestUrl": "https://researchdata.nie.edu.sg/oai",
        "archiveUrl": "https://researchdata.nie.edu.sg",
        "archiveDescription": "This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data.",
        "metadataFormat": "oai_ddi",
        "set": "NIE_Test_Set2",
        "schedule": "none",
        "status": "inActive",
        "lastHarvest": "Fri Nov 18 12:13:05 SGT 2022",
        "lastResult": "SUCCESS",
        "lastSuccessful": "Fri Nov 18 12:13:05 SGT 2022",
        "lastNonEmpty": "Fri Nov 18 12:13:05 SGT 2022",
        "lastDatasetsHarvested": "1",
        "lastDatasetsDeleted": "0",
        "lastDatasetsFailed": "0"
      }
    ]
  }
}

Best wishes,
Yikang
NIE Library

Start by examining the OAI harvesting path and the datasetversion records described in the report. Compare source and harvested records where versionstate is RELEASED, then verify that versionnumber is retained in the harvested result; the issue is done when this metadata is no longer null.