DataCite timeout: Datasets repeatedly lock on publication for >30min and process fails, second try works

What happens?

When trying to publish datasets, we repeatedly experience that they are first locked for an extraordinarily long time (up to 40 minutes) before failing the publication process. However, on the second try, the dataset is published after just a few seconds and without any issues. The error message after the first try shows that the dataset "could not be published due to a failure to register, or update the Global Identifier for the dataset or one of the files in it."

Here is a server log entry from an attempt earlier today:

[2024-10-31T13:41:47.539+0100] [Payara 6.2024.6] [WARNING] [] [edu.harvard.iq.dataverse.pidproviders.doi.datacite.DataCiteDOIProvider] [tid: _ThreadID=259 _ThreadName=__ejb-thread-pool15] [timeMillis: 1730378507539] [levelValue: 900] [[
modifyMetadata failed: Connection timed out
java.net.SocketException: Connection timed out
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:330)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at edu.harvard.iq.dataverse.pidproviders.doi.datacite.DataCiteRESTfullClient.postMetadata(DataCiteRESTfullClient.java:163)
at edu.harvard.iq.dataverse.pidproviders.doi.datacite.DOIDataCiteRegisterService.registerIdentifier(DOIDataCiteRegisterService.java:73)
at edu.harvard.iq.dataverse.pidproviders.doi.datacite.DataCiteDOIProvider.publicizeIdentifier(DataCiteDOIProvider.java:206)
at edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand.publicizeExternalIdentifier(FinalizeDatasetPublicationCommand.java:418)
at edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand.execute(FinalizeDatasetPublicationCommand.java:216)
...

[2024-10-31T13:41:47.541+0100] [Payara 6.2024.6] [WARNING] [] [edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand] [tid: _ThreadID=259 _ThreadName=__ejb-thread-pool15] [timeMillis: 1730378507541] [levelValue: 900] [[
Failed to register the identifier doi:10.11588/data/IUTCDN, or to register a file in the dataset; notifying the user(s), unlocking the dataset]

It seems like it might be connected to a timeout of the DataCite API. Outages etc. as reported here (https://status.datacite.org/) don't seem to be the problem as they appear far less often than our problems with the dataset publications.

What steps does it take to reproduce the issue?

We are unsure if others have experienced the same problems and if the described behavior is reproducible. For us, we simply try to publish a dataset via the publish button. The second try can take place just after the first one failed or later on.

When does this issue occur?

On hitting the publish button on any dataset for the first time. It happens for new datasets as well as for new versions of existing datasets. The datasets involved can have very few files with small sizes (e.g. doi:10.11588/data/NCQJJR -> 5 files with max. 90 KB each and without PID registry for each file) or be larger; we still see similar behaviour.

To whom does it occur (all users, curators, superusers)?
Superusers, curators.

What did you expect to happen?
Ideally, a publication on the first try without delays above a few minutes.

Which version of Dataverse are you using?
v. 6.3 build v6.3+10797_dls

Any related open or closed issues to this bug report?

Possibly #7393

Start with DataCiteRESTfullClient.java:163 and trace the call through DOIDataCiteRegisterService.java:73, DataCiteDOIProvider.java:206, and FinalizeDatasetPublicationCommand.java:216 and 418. Reproduce the first publication attempt using the reported timeout and inspect the existing tests around this flow. Done means a first publication attempt handles the DataCite failure without an excessive lock and the retry behavior is covered.