Feature Request/Idea: Subsetting by variable values
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
Overview of the Feature Request
It would be very useful for users to be able to subset datasets by the values of specific variables before downloading data, as for example described in the TwoRavens manual here (under "Subset"): https://guides.dataverse.org/en/5.6/user/data-exploration/tworavens.html
What kind of user is the feature intended for?
Any user downloading data.
What inspired the request?
I help administrate a fairly large repository of election result data (https://dataverse.harvard.edu/dataverse/medsl). Precinct-level results of American local, regional, and federal elections can easily reach 3 GB of text in one file, which makes the data very slow to access (a user who is looking for the results of just one statewide election still needs to download a multi-GB file with results from every race in the country, load that into software for subsetting it, and then perform the subsetting). Users can reduce the download burden by selecting specific variables, but they still have to download all of the rows. We have thought about splitting our data up and providing a variety of datasets for people to choose from, but it is very hard to anticipate exactly what data people might be after. We could for example upload one file for each state, but then users who want (say) the results of a presidential election would need to download all 50. Or we could supply each major race individually, but then users who want (say) the results of a given year's senate elections would need to download every file. I imagine there are analogous problems for large datasets about other subjects. It would be great if people could make those decisions for themselves while downloading the data.
What existing behavior do you want changed?
Only added functionality; the existing subset tool is very good.
Any brand new behavior do you want to add to Dataverse?
The option to subset by variable value when downloading data.
Any related open or closed issues to this feature request?
Not to my knowledge.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing subset tool and the linked TwoRavens manual section on “Subset” to understand current behavior. Define how variable-value filters should apply during downloads, including the user-facing selection flow and expected results for large datasets. Done means the behavior is specified, implemented, and covered by tests for filtered downloads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100