opensafely / opensafely/documentation
Document efficient analysis of large datasets
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 48
- Forks
- 10
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 17
Description
Certain study definitions can result in very large input datasets for analytical code (e.g. establishing baseline values for entire populations), especially when repeated cohort extractions are performed (e.g. monthly over a period of multiple years).
Naïve approaches to analysis of these large datasets can result in excessive RAM usage, to the detriment of successful job completion and general platform availability.
We should develop a set of general principles and recommendations for analysis of large datasets e.g.
- input minimisation
- correct datatyping
- dataframe lifecycles
And perhaps a tool for crude estimation of dataset size based on number of variables, observations, and datatypes?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the issue body and review the repository's existing documentation for guidance on large analytical datasets; no file, test, or entry point is named. Done means a documented set of recommendations covering input minimisation, datatyping, and dataframe lifecycles, with any dataset-size estimator clearly scoped.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, documentation, performance
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100