IQSS / IQSS/dataverse

Small temporary upload files are not automatically deleted after upload

Open
#12,424 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

What steps does it take to reproduce the issue?

When a file is uploaded, there are usually some upload files left in the system, in two spots.

  • dataverse.files.file.directory
  • /payara6/glassfish/domains/domain1/uploads

I think it's related to this issue, but not sure why it's not fixed in our setup: https://github.com/IQSS/dataverse/issues/10667

  • When does this issue occur?
    This issue occurs after a file is uploaded via the UI

  • Which page(s) does it occurs on?
    File upload through the UI.

  • What happens?

Files are left behind in: dataverse.files.file.directory. Here's a small sample:

-rw-r--r--. 1 dataverse dataverse 831213413 Jun 1 15:39 19e83d716d8-f6e08e442fd9
-rw-r--r--. 1 dataverse dataverse 838353413 Jun 1 15:39 19e83d75d03-bcba11fda004
-rw-r--r--. 1 dataverse dataverse 835973455 Jun 1 15:40 19e83d7a49f-4d11683f60bc
-rw-r--r--. 1 dataverse dataverse 10036 Jun 1 19:02 tempIngestSourceFile14944432732260873865.tmp
-rw-r--r--. 1 dataverse dataverse 255 Jun 1 19:02 firstpass-2728339230294518051.tab
-rw-r--r--. 1 dataverse dataverse 14065 Jun 1 19:02 tempIngestSourceFile674369604737836795.tmp
-rw-r--r--. 1 dataverse dataverse 99 Jun 1 19:02 firstpass-11771889047601741684.tab
-rw-r--r--. 1 dataverse dataverse 14853 Jun 1 19:02 tempIngestSourceFile16848639504657575746.tmp
-rw-r--r--. 1 dataverse dataverse 153 Jun 1 19:02 firstpass-1044911828866616382.tab
-rw-r--r--. 1 dataverse dataverse 15074 Jun 1 19:02 tempIngestSourceFile479945955968724125.tmp
-rw-r--r--. 1 dataverse dataverse 232 Jun 1 19:02 firstpass-12198861583407135376.tab

Sometimes these files are larger, depending on the upload.

As well as in: /payara6/glassfish/domains/domain1/uploads:

-rw-r--r--. 1 dataverse dataverse 17 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034335.tmp
-rw-r--r--. 1 dataverse dataverse 44 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034332.tmp
-rw-r--r--. 1 dataverse dataverse 34 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034329.tmp
-rw-r--r--. 1 dataverse dataverse 47 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034326.tmp
-rw-r--r--. 1 dataverse dataverse 22 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034350.tmp
-rw-r--r--. 1 dataverse dataverse 22 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034349.tmp
-rw-r--r--. 1 dataverse dataverse 45 Jun 1 19:02 upload_5df8c250_19e698dd064__7feb_00034348.tmp

These do not automatically get cleaned up by the code at all, though I thought this was fixed by https://github.com/IQSS/dataverse/pull/10867

Which version of Dataverse are you using?
6.6

Any related open or closed issues to this bug report?
https://github.com/IQSS/dataverse/pull/10867

Screenshots:

No matter the issue, screenshots are always welcome.

To add a screenshot, please use one of the following formats and/or methods described here:

Are you thinking about creating a pull request for this issue?
Help is always welcome, is this bug something you or your organization plan to fix?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing a UI upload in Dataverse 6.6 and inspect the two reported locations: dataverse.files.file.directory and /payara6/glassfish/domains/domain1/uploads. Read related issue #10667 and pull request #10867 to understand the existing cleanup behavior. Done means temporary upload, ingest, and conversion files are removed from both locations after processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.