queryverse / queryverse/StatFiles.jl
Can't load large (1gb) stata file
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 27
- Forks
- 8
- Avg merge
- 15h 51m
- Merged PRs (30d)
- 4
Description
I am trying to read stata panel datasets of Russian households and invividuals.
They can be downloaded freely after simple registration.
The houeseholds dataset is ~1gb and individuals is ~4gb. Machine has 16 gb.
Even 1gb file take too long to load (have waited > 15 minutes without any results).
using StatFiles
a = load("USER_RLMS-HSE_HH_1994_2017nt_eng.dta")
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the StatFiles.jl implementation behind the shown load("USER_RLMS-HSE_HH_1994_2017nt_eng.dta") call and reproduce the report using the linked large Stata datasets. Measure where loading spends its time and compare behavior for the approximately 1 GB and 4 GB files; done should establish whether large-file loading is functioning acceptably or identify a verified improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100