IQSS / IQSS/dataverse

Feature Request: Separation of upload limits for UI and direct

Open
#12,007 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Feature
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Overview of the Feature Request

After some discussion in https://groups.google.com/u/1/g/dataverse-community/c/ZJRAuy-1PKE there was a conclusion that :MaxFileUploadSizeInBytes limits file size in general. And while allowing user to upload small failes (let's say up to 1GB) by dataverse UI is fine, and limiting amount of data that one can upload is also fine, there is a problem with a tool to upload large data. python dvuploader should use direct upload to s3 but still is limited by value :MaxFileUploadSizeInBytes.

Simple fix on dataverse instance is to increase :MaxFileUploadSizeInBytes but this might give wrong imprssion to users that you can upload large data via UI.

What kind of user is the feature intended for?
This is intended for Depositor of large data tat would like to use tool like python dvuploader which is recommended tool for large data upload.

What existing behavior do you want changed?

MaxFileUploadSizeInBytes - should not restrict direct s3 upload.

Any brand new behavior do you want to add to Dataverse?

As we are able to diverse upload limits per type of storage by curl -X PUT -d '{"default":"2147483648","s3":"2147483648"}' http://localhost:8080/api/admin/settings/:MaxFileUploadSizeInBytes

we should be able to have options like:

  • general upload limit - current MaxFileUploadSizeInBytes
  • UI upload limit - value that will be shown to user when using upload option on the Dataverse instance
  • direct upload limit - value that will limit s3 upload via dvuploader

In the minimum version: current MaxFileUploadSizeInBytes could stay as it is and be the limit both for direct and general upload, but it would be nice to show to user different, smaller value on UI so we would not encourage users to upload large data by UI.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing MaxFileUploadSizeInBytes through the Dataverse UI upload flow, direct S3 upload path, and admin settings API. Compare the current general limit with the proposed UI and direct-upload limits; done means the configuration and displayed UI value can be separated without incorrectly restricting direct uploads.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
api, backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.