Setting files-not-accessible-by-dataverse is not being utilized in file add when using Trusted Remote Storage
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Dataverse tries to access the file in Trusted Remote Storage when attaching a TRS-located file to the dataset. This is because Dataverse is trying to gauge file size using HTTP HEAD which fails when target system idoes not return Content-Length header.
My goal is to link to a dataset landing page in a target system, instead of actual file. When storageIdentifier in JSON payload points to an actual file which can be accessed by Dataverse, I do not get an error. When the storageIdentifier points to a landing page, I get the following error. I'm running Dataverse 6.10.1 .
Configuration guide suggests that files-not-accessible-by-dataverse should be false when using landing page instead of actual file. I tested this also with setting files-not-accessible-by-dataverse to true and this change had no effect. Regardless of the setting, I get the following error:
#|2026-04-16T16:03:13.803+0300|SEVERE|Payara 7.2026.3|edu.harvard.iq.dataverse.api.errorhandlers.ThrowableHandler|_ThreadID=102;_ThreadName=http-thread-pool::jk-connector(3);_TimeMillis=1776344593803;_LevelValue=1000;|
_status="ERROR";_code=500;_message="Internal server error. More details available at the server logs.";_incidentId="1cf5601d-2d55-4174-abbf-f03761f2d41e";_internalError="ArrayIndexOutOfBoundsException";_requestUrl="https://xxxxx/api/v1/datasets/:persistentId/add?persistentId=doi:10.5072/FK2/BRLYML";_requestMethod="POST"||#]
[#|2026-04-16T16:03:13.803+0300|SEVERE|Payara 7.2026.3|edu.harvard.iq.dataverse.api.errorhandlers.ThrowableHandler|_ThreadID=102;_ThreadName=http-thread-pool::jk-connector(3);_TimeMillis=1776344593803;_LevelValue=1000;|
java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
at edu.harvard.iq.dataverse.dataaccess.RemoteOverlayAccessIO.retrieveSizeFromMedia(RemoteOverlayAccessIO.java:172)
My TRS config is:
-Ddataverse.files.vault.type=remote
-Ddataverse.files.vault.label=Vault
-Ddataverse.files.vault.base-url=https://internalurl.xy/records
-Ddataverse.files.vault.remote-store-name=Remote dataset
-Ddataverse.files.vault.base-store=file
-Ddataverse.files.vault.upload-out-of-band=true
-Ddataverse.files.vault.public=true
-Ddataverse.files.vault.ingestsizelimit=0
-Ddataverse.files.vault.download-redirect=true
-Ddataverse.files.vault.files-not-accessible-by-dataverse=true
-Ddataverse.files.vault.remote-store-url=https://internalurl.xy/info
-Ddataverse.files.vault.remote-store-name=Vault
When Dataverse cannot reach target server at all, adding files works as expected. Error happens in case when user tries to link to a landing page and the target system does not return Content-Length header because retrieveSizeFromMedia() assumes that target system response always contains Content-Length header.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with RemoteOverlayAccessIO.java at retrieveSizeFromMedia(), especially line 172, and trace the dataset add API path that invokes it for Trusted Remote Storage. Compare behavior when files-not-accessible-by-dataverse is true and when the remote response lacks Content-Length. Done means adding a landing-page reference no longer raises ArrayIndexOutOfBoundsException and the configured setting is respected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100