IQSS / IQSS/dataverse

Spike: Investigate challenges with how Dataverse software handles shapefiles

Open
#8,816 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: File Upload & Handling
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Challenges with how the Dataverse software handles shapefiles was mentioned in a GitHub issue at https://github.com/IQSS/dataverse/issues/6873#issuecomment-624804056. My questions then were more about computational reproducibility. But this has come up again because a depositor I'm trying to help has concerns with how this functionality is complicating their upload of a lot of data onto the Harvard Dataverse Repository.

The depositor, who's using Dataverse APIs to upload files that are not on their computer (I think the files are on an AWS server), may or may not be able to detect and double zip all shapefiles in order to prevent the Dataverse software from zipping the shapefiles when they're uploaded to the repository. I'll ask the depositor if they can do this.

But:

  • When uploading files using the Dataverse APIs, is it possible to tell the Dataverse software not to put shapefiles in a .zip file?
  • As @qqmyers mentioned during a meeting this morning, can the repository's ingest settings be temporarily changed so that the Dataverse software doesn't zip the shapefiles in this depositor's uploads?

For more context, the email conversation is in IQSS's support email system at https://help.hmdc.harvard.edu/Ticket/Display.html?id=322323 and the data is from Redistricting Data Hub

More broadly, I think more research should be done about the value of the Dataverse software's handling of shapefiles, including the questions and discussion in the GitHub issue comment at https://github.com/IQSS/dataverse/issues/6873#issuecomment-624804056

The issue at https://github.com/IQSS/dataverse/issues/7352 might also be related.

Having ways for depositors to learn about this behavior before they start uploading would be helpful. This behavior is documented only in the Developer guides (https://guides.dataverse.org/en/6.2/developers/geospatial.html) and not in the User Guides or in the UI, although it's referenced in the User Guides.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the shapefile handling discussion in issues 6873 and 7352, the API upload behavior, and the developer and user guide sections linked in the issue. Determine whether the current ingest behavior can be controlled for these uploads and what depositor-facing documentation or UI changes are needed; done means the challenges and actionable next steps are documented.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.