queryverse / queryverse/ReadStat.jl
Streaming iterator read
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 79
- Forks
- 18
- Avg merge
- 22h 31m
- Merged PRs (30d)
- 3
Description
One of the most important features is to be able to read a chunk of SAS and then process it because in the wild there are a lot of large SAS datasets which are 50G~100G in size and they can't fit into memory.
What's stopping wider move away from SAS I think is due to not being able to read the data in chunks and then process it and then read the next chunk.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files, tests, or entry points. Start by locating the existing SAS reader in ReadStat.jl and determine how it currently loads data; define chunk boundaries and the processing API, then verify that 50G–100G SAS datasets can be handled without fitting in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100