CouncilDataProject / CouncilDataProject/cdp-data
Prototype delayed dask dataframes for better computation scaling
Open
enhancement
- Dominant language
- Jupyter Notebook
- Stars
- 5
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
Currently all computation is multithreaded by default and that doesn't leave a lot of room to the user in terms of deciding how and when they want to actually gather data or compute a result.
The datasets module can easily be switched over to dask for data gathering and caching
The keywords module should be looked at for how to use dask dataframes for just-in-time compute and out-of-memory compute.
Contributor guide
Assessment
This issue has not been assessed yet.