IQSS / IQSS/dataverse

6.1+/EZID: publish dataset fails when metadata contains ampersands

Open
#10,830 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

What steps does it take to reproduce the issue?

  1. have an instance of Dataverse 6.1 or higher using EZID for DOI minting
  2. create a new dataset that has metadata containing an (unescaped) ampersand in it (i.e. & instead of &).
  3. attempt to publish the dataset
  • When does this issue occur?

    • the error occurs when you try to publish the dataset
  • Which page(s) does it occurs on?

    • edit dataset & view dataset
  • What happens?

    • publication fails
    • log messages like what's in server.log below
    • in Firefox and Chrome, the dataset page won't load because of an unescaped character
    • any existing citation metadata may become corrupted (i'm less certain about about this, but ran into it) (edit: this may be the bug fixed by #10797)
  • To whom does it occur (all users, curators, superusers)?

    • curators/superusers
  • What did you expect to happen?

    • either publication to succeed or to have a clear error message

Which version of Dataverse are you using?

6.1

Any related open or closed issues to this bug report?

#3328, #3845, #7611

Are you thinking about creating a pull request for this issue?

not at this point; existing workaround to replace ampersands with "and" will work for us

server.log
[2024-09-09T14:16:45.442-0700] [Payara 6.2023.9] [WARNING] [] [edu.harvard.iq.dataverse.DOIEZIdServiceBean] [tid: _ThreadID=109 _ThreadName=http-thread-pool::jk-connector(4)] [timeMillis: 1725916605442] [levelValue: 900]
 [[ modifyMetadata failed]]

[2024-09-09T14:16:45.442-0700] [Payara 6.2023.9] [WARNING] [] [edu.harvard.iq.dataverse.DOIEZIdServiceBean] [tid: _ThreadID=109 _ThreadName=http-thread-pool::jk-connector(4)] [timeMillis: 1725916605442] [levelValue: 900]
 [[
  String edu.ucsb.nceas.ezid.EZIDException: bad request - error="ValidationError({'datacite': ['Metadata validation error: XML parse error: EntityRef: expecting \';\', line 6, column 40 (<string>, line 6). metadata="<?xml version="1.0" encoding="UTF-8"?>\n<resource xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://s
chema.datacite.org/meta/kernel-4/metadata.xsd"\n          xmlns="http://datacite.org/schema/kernel-4"\n   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n    <identifier identifierType="DOI">10.60503/D3/FFXLIL</identifier>\n    <creators><creator><creatorName>S&P Global</creatorName></creator></creators>\n    <titles>\n        <title>RateWatch Scholar</title>\n    </titles>\n    <publisher>UC Berkeley Library Dataverse</publisher>\n    <publicationYear>2024</publicationYear>\n    <resourceType resourceTypeGeneral="Dataset"/>\n \n    <descriptions>\n        <description descriptionType="Abstract">RateWatch Scholar offers the academic community information for U.S. financial institutions for research and analysis. Data covers over 96,000 branch locations, depending on time period and data type, all provided voluntarily. Data is gathered from institutions of all types and sizes, including banks, credit unions, savings and loan associations, etc. The RateWatch Historical data sets focus on retail products offered to the general public. Deposit rates data: 2001 - 2020 Loan rates data: 2022 Fee data:</description>\n    </descriptions>\n    <contributors><contributor contributorType="ContactPerson"><contributorName>Library Data Services Program</contributorName><affiliation>(UC Berkeley)</affiliation></contributor></contributors>\n</resource>"']})" Metadata: {'datacite': '<?xml version="1.0" encoding="UTF-8"?>\n'              '<resource '              'xsi:schemaLocation="http://datacite.org/schema/kernel-4 '              'http://schema.datacite.org/meta/kernel-4/metadata.xsd"\n'              '          xmlns="h
ttp://datacite.org/schema/kernel-4"\n'              '          '              'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n'              '    <identifier '              'identifierType="DOI">10.60503/D3/FFXLIL</identifier>\n'              '    <creators><creator><creatorName>S&P '              'Global</creatorName></creator></creators>\n'              '    <titles>\n'              '        <title>RateWatch Scholar</title>\n'              '    </titles>\n'              '    <publisher>UC Berkeley Library Dataverse</publisher>\n'              '    <publicationYear>2024</publicationYear>\n'              '    <resourceType resourceTypeGeneral="Dataset"/>\n'              '    \n'              '    <descriptions>\n'              '        <description descriptionType="Abstract">RateWatch '              'Scholar offers the academic community information for U.S. '              'financial institutions for research and analysis. Data covers '              'over 96,000 branch locations, depending on time period and data '              'type, all provided voluntarily. Data is gathered from '              'institutions of all types and sizes, including banks, credit '              'unions, savings and loan associations, etc. The RateWatch '              'Historical data sets focus on retail products offered to the '              'general public. Deposit rates data: 2001 - 2020 Loan rates data: '              '2022 Fee data:</description>\n'              '    </descriptions>\n'              '    <contributors><contributor '              'contributorType="ContactPerson"><contributorName>Library Data '              'Services Program</contributorName><affiliation>(UC '              'Berkeley)</affiliation></contributor></contributors>\n'              '</resource>',  'datacite.resourcetype': 'Dataset'}]]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the publish flow and the DOIEZIdServiceBean named in the server log, then reproduce publication using metadata containing an unescaped ampersand. Confirm that publication succeeds or reports a clear error, and that the dataset page and citation metadata remain usable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.