IQSS / IQSS/dataverse

Setting files-not-accessible-by-dataverse is not being utilized in file add when using Trusted Remote Storage

Open
#12,339 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Dataverse tries to access the file in Trusted Remote Storage when attaching a TRS-located file to the dataset. This is because Dataverse is trying to gauge file size using HTTP HEAD which fails when target system idoes not return Content-Length header.

My goal is to link to a dataset landing page in a target system, instead of actual file. When storageIdentifier in JSON payload points to an actual file which can be accessed by Dataverse, I do not get an error. When the storageIdentifier points to a landing page, I get the following error. I'm running Dataverse 6.10.1 .

Configuration guide suggests that files-not-accessible-by-dataverse should be false when using landing page instead of actual file. I tested this also with setting files-not-accessible-by-dataverse to true and this change had no effect. Regardless of the setting, I get the following error:

#|2026-04-16T16:03:13.803+0300|SEVERE|Payara 7.2026.3|edu.harvard.iq.dataverse.api.errorhandlers.ThrowableHandler|_ThreadID=102;_ThreadName=http-thread-pool::jk-connector(3);_TimeMillis=1776344593803;_LevelValue=1000;|
_status="ERROR";_code=500;_message="Internal server error. More details available at the server logs.";_incidentId="1cf5601d-2d55-4174-abbf-f03761f2d41e";_internalError="ArrayIndexOutOfBoundsException";_requestUrl="https://xxxxx/api/v1/datasets/:persistentId/add?persistentId=doi:10.5072/FK2/BRLYML";_requestMethod="POST"||#]

[#|2026-04-16T16:03:13.803+0300|SEVERE|Payara 7.2026.3|edu.harvard.iq.dataverse.api.errorhandlers.ThrowableHandler|_ThreadID=102;_ThreadName=http-thread-pool::jk-connector(3);_TimeMillis=1776344593803;_LevelValue=1000;|
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO.retrieveSizeFromMedia(RemoteOverlayAccessIO.java:172)

My TRS config is:


-Ddataverse.files.vault.type=remote
-Ddataverse.files.vault.label=Vault
-Ddataverse.files.vault.base-url=https://internalurl.xy/records
-Ddataverse.files.vault.remote-store-name=Remote dataset
-Ddataverse.files.vault.base-store=file
-Ddataverse.files.vault.upload-out-of-band=true
-Ddataverse.files.vault.public=true
-Ddataverse.files.vault.ingestsizelimit=0
-Ddataverse.files.vault.download-redirect=true
-Ddataverse.files.vault.files-not-accessible-by-dataverse=true
-Ddataverse.files.vault.remote-store-url=https://internalurl.xy/info
-Ddataverse.files.vault.remote-store-name=Vault

When Dataverse cannot reach target server at all, adding files works as expected. Error happens in case when user tries to link to a landing page and the target system does not return Content-Length header because retrieveSizeFromMedia() assumes that target system response always contains Content-Length header.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with RemoteOverlayAccessIO.java at retrieveSizeFromMedia(), especially line 172, and trace the dataset add API path that invokes it for Trusted Remote Storage. Compare behavior when files-not-accessible-by-dataverse is true and when the remote response lacks Content-Length. Done means adding a landing-page reference no longer raises ArrayIndexOutOfBoundsException and the configured setting is respected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.