IQSS / IQSS/dataverse

Feature Request/Idea: Subsetting by variable values

Open
#8,371 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Feature: API Feature: DOI & Handle Type: Feature User Role: API User
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Overview of the Feature Request
It would be very useful for users to be able to subset datasets by the values of specific variables before downloading data, as for example described in the TwoRavens manual here (under "Subset"): https://guides.dataverse.org/en/5.6/user/data-exploration/tworavens.html

What kind of user is the feature intended for?
Any user downloading data.

What inspired the request?
I help administrate a fairly large repository of election result data (https://dataverse.harvard.edu/dataverse/medsl). Precinct-level results of American local, regional, and federal elections can easily reach 3 GB of text in one file, which makes the data very slow to access (a user who is looking for the results of just one statewide election still needs to download a multi-GB file with results from every race in the country, load that into software for subsetting it, and then perform the subsetting). Users can reduce the download burden by selecting specific variables, but they still have to download all of the rows. We have thought about splitting our data up and providing a variety of datasets for people to choose from, but it is very hard to anticipate exactly what data people might be after. We could for example upload one file for each state, but then users who want (say) the results of a presidential election would need to download all 50. Or we could supply each major race individually, but then users who want (say) the results of a given year's senate elections would need to download every file. I imagine there are analogous problems for large datasets about other subjects. It would be great if people could make those decisions for themselves while downloading the data.

What existing behavior do you want changed?
Only added functionality; the existing subset tool is very good.

Any brand new behavior do you want to add to Dataverse?
The option to subset by variable value when downloading data.

Any related open or closed issues to this feature request?
Not to my knowledge.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing subset tool and the linked TwoRavens manual section on “Subset” to understand current behavior. Define how variable-value filters should apply during downloads, including the user-facing selection flow and expected results for large datasets. Done means the behavior is specified, implemented, and covered by tests for filtered downloads.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.