IQSS / IQSS/dataverse

When direct upload is enabled, prevent the default upload API from accepting giant files

Open
#9,274 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Our recommended way of supporting users who need to upload large volumes of data is to assign storage volumes ("stores") with increased size limits and direct upload enabled.
This generally works great, but leaves a serious vulnerability: a user may still try to upload their gigantic files via the normal/"classic" API (unknowingly, or if they miss the "-direct" option in DvUploader), and it will attempt to accept the upload, potentially flooding the temp space and otherwise subjecting the system to all the problems associated with using that API for large volumes of data.

I'd like to add logic to that API to enforce the default, installation-wide file size limit when it's being used to upload a file into a store with direct upload enabled; instead of the increased limit associated with the store.

Ideally, I would like to be able to reject that upload as soon as it reaches that limit - i.e. before the application saves the entire thing in java.io.tmpdir... But that may or may not be possible.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the default or “classic” upload API and how it handles stores with direct upload enabled; compare that path with the DvUploader “-direct” option and the installation-wide limit. The change is done when classic uploads to such stores are rejected at the default limit, ideally before the full file is saved in java.io.tmpdir.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.