IQSS / IQSS/dataverse

Harvesting: processing oai_dc:identifiers without Global Id protocol prefix: defaults to hdl in a potentially very inefficient way

Open
#10,429 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: Harvesting Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

[the paragraph that follows need to be edited some more, as it still contains some inaccuracies; will do once I have a sec. but, to emphasize, this is a fairly exotic/uncommon issue]

Not a very common problem, but was the case with SRDA from #7624: their OAI server is supplying the record identifiers like this: 10.6141/TW-SRDA-AN010012-1 - i.e., without the doi: prefix. This is a valid doi, and resolving it, as in https://doi.org/10.6141/TW-SRDA-AN010012-1, works. However our code appears to default to hdl: (!) - and that doesn't work of course. We just need to make this configurable on the client level, which protocol to default to when the prefix is not supplied.

Upon further investigation:
It does not just default to handles; ImportGenericServiceBean has this code:

for (String otherId : otherIds) {
   try {
      HandleResolver hr = new HandleResolver();
      hr.resolveHandle(otherId);
      return HandlenetServiceBean.HDL_PROTOCOL + ":" + otherId;
    } catch (HandleException e) {
       logger.fine("Not a valid handle: " + e.toString());
    }
}

i.e., it will try to resolve every "other identifier" - i.e, identifiers that cannot be unambiguously identified as either DOI or Handle - as a handle. In IQSS production however, it makes harvesting impossible - because the attempts to resolve above take a long time (before timing out) - probably because of some firewall.

We want to be able to turn the lookup mechanism off (but we want to add it in place by default, since somebody must have included it for a reason), by being able to specify either "doi:" or "handle:", as the prefix to default to? - should be straightforward if we make this API-only functionality. (once again, this is a fairly exotic condition; never happens when harvesting from another dataverse)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading ImportGenericServiceBean and the handling around otherIds, including HandleResolver and HandlenetServiceBean. Reproduce harvesting with an unprefixed identifier to observe the handle lookup behavior and delay. Done means the client-level API configuration can disable lookup or select a default doi: or handle: prefix while retaining the existing default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.