6.1+/EZID: publish dataset fails when metadata contains ampersands
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
What steps does it take to reproduce the issue?
- have an instance of Dataverse 6.1 or higher using EZID for DOI minting
- create a new dataset that has metadata containing an (unescaped) ampersand in it (i.e.
&instead of&). - attempt to publish the dataset
-
When does this issue occur?
- the error occurs when you try to publish the dataset
-
Which page(s) does it occurs on?
- edit dataset & view dataset
-
What happens?
- publication fails
- log messages like what's in server.log below
- in Firefox and Chrome, the dataset page won't load because of an unescaped character
- any existing citation metadata may become corrupted (i'm less certain about about this, but ran into it) (edit: this may be the bug fixed by #10797)
-
To whom does it occur (all users, curators, superusers)?
- curators/superusers
-
What did you expect to happen?
- either publication to succeed or to have a clear error message
Which version of Dataverse are you using?
6.1
Any related open or closed issues to this bug report?
#3328, #3845, #7611
Are you thinking about creating a pull request for this issue?
not at this point; existing workaround to replace ampersands with "and" will work for us
server.log
[2024-09-09T14:16:45.442-0700] [Payara 6.2023.9] [WARNING] [] [edu.harvard.iq.dataverse.DOIEZIdServiceBean] [tid: _ThreadID=109 _ThreadName=http-thread-pool::jk-connector(4)] [timeMillis: 1725916605442] [levelValue: 900]
[[ modifyMetadata failed]]
[2024-09-09T14:16:45.442-0700] [Payara 6.2023.9] [WARNING] [] [edu.harvard.iq.dataverse.DOIEZIdServiceBean] [tid: _ThreadID=109 _ThreadName=http-thread-pool::jk-connector(4)] [timeMillis: 1725916605442] [levelValue: 900]
[[
String edu.ucsb.nceas.ezid.EZIDException: bad request - error="ValidationError({'datacite': ['Metadata validation error: XML parse error: EntityRef: expecting \';\', line 6, column 40 (<string>, line 6). metadata="<?xml version="1.0" encoding="UTF-8"?>\n<resource xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://s
chema.datacite.org/meta/kernel-4/metadata.xsd"\n xmlns="http://datacite.org/schema/kernel-4"\n xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n <identifier identifierType="DOI">10.60503/D3/FFXLIL</identifier>\n <creators><creator><creatorName>S&P Global</creatorName></creator></creators>\n <titles>\n <title>RateWatch Scholar</title>\n </titles>\n <publisher>UC Berkeley Library Dataverse</publisher>\n <publicationYear>2024</publicationYear>\n <resourceType resourceTypeGeneral="Dataset"/>\n \n <descriptions>\n <description descriptionType="Abstract">RateWatch Scholar offers the academic community information for U.S. financial institutions for research and analysis. Data covers over 96,000 branch locations, depending on time period and data type, all provided voluntarily. Data is gathered from institutions of all types and sizes, including banks, credit unions, savings and loan associations, etc. The RateWatch Historical data sets focus on retail products offered to the general public. Deposit rates data: 2001 - 2020 Loan rates data: 2022 Fee data:</description>\n </descriptions>\n <contributors><contributor contributorType="ContactPerson"><contributorName>Library Data Services Program</contributorName><affiliation>(UC Berkeley)</affiliation></contributor></contributors>\n</resource>"']})" Metadata: {'datacite': '<?xml version="1.0" encoding="UTF-8"?>\n' '<resource ' 'xsi:schemaLocation="http://datacite.org/schema/kernel-4 ' 'http://schema.datacite.org/meta/kernel-4/metadata.xsd"\n' ' xmlns="h
ttp://datacite.org/schema/kernel-4"\n' ' ' 'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n' ' <identifier ' 'identifierType="DOI">10.60503/D3/FFXLIL</identifier>\n' ' <creators><creator><creatorName>S&P ' 'Global</creatorName></creator></creators>\n' ' <titles>\n' ' <title>RateWatch Scholar</title>\n' ' </titles>\n' ' <publisher>UC Berkeley Library Dataverse</publisher>\n' ' <publicationYear>2024</publicationYear>\n' ' <resourceType resourceTypeGeneral="Dataset"/>\n' ' \n' ' <descriptions>\n' ' <description descriptionType="Abstract">RateWatch ' 'Scholar offers the academic community information for U.S. ' 'financial institutions for research and analysis. Data covers ' 'over 96,000 branch locations, depending on time period and data ' 'type, all provided voluntarily. Data is gathered from ' 'institutions of all types and sizes, including banks, credit ' 'unions, savings and loan associations, etc. The RateWatch ' 'Historical data sets focus on retail products offered to the ' 'general public. Deposit rates data: 2001 - 2020 Loan rates data: ' '2022 Fee data:</description>\n' ' </descriptions>\n' ' <contributors><contributor ' 'contributorType="ContactPerson"><contributorName>Library Data ' 'Services Program</contributorName><affiliation>(UC ' 'Berkeley)</affiliation></contributor></contributors>\n' '</resource>', 'datacite.resourcetype': 'Dataset'}]]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the publish flow and the DOIEZIdServiceBean named in the server log, then reproduce publication using metadata containing an unescaped ampersand. Confirm that publication succeeds or reports a clear error, and that the dataset page and citation metadata remain usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100