IQSS / IQSS/dataverse

Zipper assumes single S3 profile

Open
#8,806 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

HDC
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Looking at the code for other reasons I saw https://github.com/IQSS/dataverse/blob/46860665574a2cd796095f6fc8c5ce96f16f0fc1/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java#L562-L575 which appears to remove store names from the locations and replace them with the store type (e.g. s3 or file in most cases). Looking at https://github.com/IQSS/dataverse/pull/6986/files#diff-7770650e35b5e54de11afe7c79f54715bdd20e120a446414a02db4a881c77438 , it appears that this location is then used to find the files for the Zipper util.

I did not test/verify, but my guess is that this works in most cases as all file stores provide an absolute path and s3 stores include the bucket name. However, by ignoring the specific store's info, with S3, if a store uses a different profile, or has a different endpoint URL, or other params, the file won't be retrievable.

This would affect any installation that uses s3 stores from different places that configures the Zipper (e.g. Harvard if/when a Globus-enabled S3 store at Harvard is configured along with other AWS s3 stores.)

Which version of Dataverse are you using?
5.11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with FileDownloadServiceBean.java at lines 562-575 and the Zipper-related code shown in pull request 6986. Verify how store locations are transformed and whether a zipper configured with S3 stores using different profiles or endpoint URLs can retrieve each file. Done means files from distinct S3 configurations remain retrievable.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.