queryverse / queryverse/ReadStat.jl

Reading `.dta` with value labels

Open
#74 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
79
Forks
18
Avg merge
22h 31m
Merged PRs (30d)
3

Description

As you know, Stata basically stores value-labeled data as a vector of integers or doubles, not necessarily an ordered sequence starting at 1, and a Dict going from Int => String.

Accessing the string values, which we generally care the most about, is hard with ReadStat. You have to

  1. Use ReadStat not StatFiles to access the internal fields of the Stata File
  2. Construct the DataFame from the data and header fields
    3 . Use the value_label_dict field to perform the replacement
  3. Use get on the DataValue elements of the array

This is not the most user friendly thing.

There isn't a great solution for this in Julia as we dont have a CategoricalArray equivalent where the base dict maps arbitrary types to strings. So converting to categorical array will drop the underlying integers, which are useful to keep due to inter-operability.

haven in R recently made a change with how this is handled with the <dbl+lbl> vector type. Though working with it is a bit of a pain, see here.

I can email a data-set to someone with an MWE for more information.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing how ReadStat exposes Stata value-label fields and how StatFiles currently constructs the DataFrame, including value_label_dict and DataValue access. Compare the issue's described workflow with haven's <dbl+lbl> approach; done should provide a user-friendly way to access labels while preserving the underlying integer or double values.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.